Title and tags

You may remember from my last blog post that I chose to have column on the Events db table named tags. My initial thinking has been that most people either would want to search on the title of the event, or some other metadata about the event. I chose to go with tags as people are used to adding these to their posts on social media, and would be comfortable using them to add searchable values to their events.
The next question then, is how should these tags be represented in the database? I need the tags to be easily searchable, with fast querying in the database, and I need every event to be able to hold several tags. One solution would be to have a tags table with a column that holds a string value that is the tag, and a tag ID. Then have a through table that points to a tag and an event in the same row, to register which events hold which tags. However, this would require expensive joins to query for specific tags. A successful application, holding thousands of searchable events, would make for some inefficient querying this way.
So I need locality of the tags, and I need to be able to index said tags as effectively as possible. Because I don’t need tags to be normalised in the database (I don’t care about aggregation, etc), I can hold all of the tags in a single column on each entry.

PostgreSQL has an index type which helps immensely with lookups in arrays called GIN (Generalized inverted index). GIN works really well on JSONB, array types and full text search. It is really effective in speeding up searches on these data types in a way that B-tree indexes cannot, and is especially effective if you are using the @> operator. It does come with the added cost of extra overhead on writes, but as I expect that there will be considerably many more searches for events than events being created or updated, I am okay with this overhead. There are also ways to mitigate some of the overhead.
Defining the column on the events table in go, using GORM looks like this:

Tags         pq.StringArray `gorm:"type:text[];index:tags_idx,type:GIN"`

Here we tell GORM about the datatype using pq.StringArray and define the column to be a text array with type:text[]. I then specify the index as well as the indexing type using index:tags_idx,type:GIN.

The title is more simply defined:

Title        string         `gorm:"not null"`

Querying events

So now that we have defined our tags and title column, we can look at querying events. Having in mind that a user will be writing their search criteria in a single search bar, it becomes apparent that we need to make our lookups using the same values for both title and tags. This means that the words written in a search could either be a list of tags or an event title. To facilitate this, the words need to be both comma separated for the tags and with spaces for the title. Assuming that the api takes a list of words delimited by commas in the query parameters, we can query the database like so:

type EventSearchItemDTO struct {
    ID               uuid.UUID `json:"id"`
    Title            string    `json:"title"`
    FormattedAddress string    `json:"address"`
}

func (es *EventStore) Get(c echo.Context) error {
    var events = []EventSearchItemDTO{}
    var result *gorm.DB

    params := c.QueryParams()
    tags := params.Get("tags")
    title := strings.ReplaceAll(tags, ",", " ")
    //ILIKE makes the LIKE case insensitive
    result = es.db.Model(&model.Event{}).Where("tags @> ? OR title ILIKE ?", "{"+tags+"}", "%"+title+"%").Select("id, title, formatted_address").Find(&events)

    if result.Error != nil {
        return c.String(http.StatusNotFound, "Not Found")
    }
    return c.JSON(http.StatusOK, events)
}

The resulting database query, searching for “cool party” looks like this:

SELECT id, title, formatted_address FROM "events" WHERE tags @> '{cool,party}' OR title ILIKE '%cool party%'

A thing to note, is that I’m currently calling the search criteria for tags, and then deducting the title search criteria from that, changing commas to spaces. Since it’s both things we’re searching for, the query parameter should probably not be named tags.

Improvements

Currently, searching for event title is not exactly efficient. If a user misspells a word or uses other derivatives of it (for example writing days instead of day), they would not get a hit on an event that would otherwise be a match. So we need a better search functionality for the title. But what about getting hits on some text in the description of the event? Maybe the person even wants to search for events on a specific address?
One potential solution is to add an extra column in the database that converts all relevant columns into a tsvector for full-text search. This brings a new set of complications though. The tags that I have implemented do not need to be searched for derivatives, making them unsuitable for tsvector search. Full text search also means that we have to think about settings weights on values in order to determine the most relevant matches and displaying them first.
An alternative I have been thinking about, is getting rid of the tags, and using categories for filtering instead, and combining it with full text search on all columns. If you have any input or suggestions I would love to hear from you in the comments.

Final thoughts

In this post, we explored how to query events using a combination of tags and event titles, leveraging GIN indexing and the less complicated ILIKE operator. We also discussed alternative approaches to enhance search functionality. If you find these problems interesting or want to learn more about full text search, I recommend checking out pganalyze’s article Understanding Postgres GIN Indexes.
Another method to sort results by relevance is to consider the proximity of the event to the user. In my next post, I will discuss how to perform distance calculations directly in PostgreSQL.
You can access my code and review it in detail on my GitHub repo.

The Model

I want to expose a simple API that tracks events - public and private - as well as users and the events they have created and/or are attending. Simple as that. I do, however, want to start off by focusing on the events part, so let’s take a look at what I want an “event” to entail:

• Title

• Description

• A banner/image

• Start datetime

• End datetime

• Address

• Geolocation for distance calculation purposes

Pretty simple.

We need to be able to hold a little more information about the event though.

• The event should have one or more owners

• Users should be able to subscribe to an event - Both as attending and non-attending

• For search and filtering purposes the owner of an event should be able to add tags

• The owner should be able to make the event private, making it invisible to the public

• The owner should be able to disable the event - effectively cancelling it

With GORM you define a database table as a go struct. Representing the table would therefore look something like this:

package model

import (
    "time"

    "github.com/codingsince1985/geo-golang"
    "github.com/lib/pq"
)

type Event struct {
    Model
    Title            string         `gorm:"not null;index:title_idx,type:GIN"`
    Description      string         `gorm:"default:NULL"`
    Banner_url       string         `gorm:"default:NULL"`
    Owners           []User         `gorm:"many2many:event_owners"`
    Is_private_event bool           `gorm:"default:false"`
    Time_start       time.Time      `gorm:"required"`
    Time_end         time.Time      `gorm:"default:NULL;check:time_end > time_start"`
    Address          *geo.Address   `gorm:"embedded"`
    Geolocation      *geo.Location  `gorm:"embedded"`
    Tags             pq.StringArray `gorm:"type:text[];index:tags_idx,type:GIN"`
    Is_enabled       bool           `gorm:"default:true"`
    Subscribers      []User         `gorm:"many2many:event_subscribers"`
}

Notice the tags for each field in the struct, which start with "gorm:". GORM’s migrate functionality interprets these tags and applies them as constraints on the corresponding database columns when creating the table. This allows us to define a database table using just Go code. Pay attention to the Address and Geolocation fields, as well as the Tags field. The Address and Geolocation types refer to structs from the imported package geo-golang, a geocode package. The Address struct includes fields like "Street" and "HouseNumber", and by using the gorm tag "embedded", we instruct GORM to include all fields in the struct as individual columns in the database table. The Location struct contains fields for latitude and longitude.

One downside of using embedded structs is the inability to apply database constraints or other tags to individual fields, which can make them challenging to work with. In such cases, you might prefer to define the structs yourself instead of importing them from external packages.

The Tags field is a little different. I need to be able to hold several tags across several events and still be able to query them fast. I chose to use a StringArray with GIN indexing, a choice I will elaborate on in a future post about searching and filtering events and writing the appropriate queries.

The Subscribers field is defined as a many-to-many relation with the though-table being called “event_subscribers”. This is a simple solution, and I want to define my own table in the future, so it can hold information such as attending/not attending/maybe attending, and whether the user wants to receive updates about the event they subscribed to. For now, we will define it this way.

You might have noticed the Model field in the beginning of the Event struct. The struct is inheriting from the Model struct, which I have defined in a different file in the same package. The struct holds a unique ID for the row as well as some metadata. GORM has it’s own model struct which you would normally inherit, but it has it’s limitations and, as discussed earlier, I cannot use struct tags on those fields as it is a struct imported from an external package. The gorm.Model looks as such:

type Model struct {
    ID        uint `gorm:"primarykey"`
    CreatedAt time.Time
    UpdatedAt time.Time
    DeletedAt DeletedAt `gorm:"index"`
}

I do not wish to use unsigned integers as the ID for the database tables, as UUID enables you to uniquely identifying any row in any table. It also makes it near impossible to guess an ID. Therefore I have defined my own base model:

package model

import (
    "database/sql"
    "time"

    "github.com/google/uuid"
)

type Model struct {
    ID        uuid.UUID    `gorm:"<-:create;type:uuid;primary_key;default:uuid_generate_v4()" param:"id" validate:"omitempty,uuid4"`
    CreatedAt time.Time    `gorm:"autoCreateTime:milli"`
    UpdatedAt time.Time    `gorm:"autoUpdateTime:milli"`
    DeletedAt sql.NullTime `gorm:"index"`
}

Here I make sure a new row always has a UUID generated when first created, as well as other tags for validation and linking it to the request parameter “id”. I also made sure to explicitly define how I want time to be stored in the rows. Specifically in milliseconds. I am going to go into more detail about other struct tags than gorm in a later post.

Creating the endpoints

Now let’s have a look at how to expose an api as simply as possible using Echo.

func main() {
    //Connect to db and return a db object
    db, err := dbmodule.Connect()
    if err != nil {
        panic(err)
    }
    //Migrate the registered structs
    dbmodule.Migrate(db)
    //Create an instance of echo 
    e := echo.New()
    //Define routing
    v1 := e.Group("/v1")
    event := v1.Group("/event")
    event.POST("/", func(c echo.Context) error {
        //Create event logic
    })
    event.GET("/", func(c echo.Context) error {
        //Get event logic
    })
    event.GET("/:id", func(c echo.Context) error {
        //Get event by id logic
    })
    event.PUT("/:id", func(c echo.Context) error {
        //Update event logic
    })
    event.POST("/:id/image", func(c echo.Context) error {
        //Upload image to event logic
    })
    event.DELETE("/:id", func(c echo.Context) error {
        //Delete event logic
    })

    e.Logger.Fatal(e.Start(":1323"))
}

When run, this should be enough to create an Echo server which listens on port 1323. I have boiled it down quite a bit from my actual implementation. As you can see this does not show how I handle the database connection nor the migration. I doesn’t show my implementation of each endpoint, and I register the routes differently. However, this should be enough to get the ball rolling.

Final thoughts

We have looked at how we can use Go structs with tags to represent tables with GORM. We have also had a brief look at how we can expose an api very quickly, using Echo framework. Some of the code shown above has been simplified to keep the initial definition of the system as simple as possible. For instance, the Event struct in my actual implementation includes more tags for validation and binding purposes. I have also implemented automatic TLS certification and request validation, among other features.
You can access my code and review it in detail on my GitHub repo.

In my last post I talked about some of the requirements of the application and the fact that I really wanted to try to build the server side in Golang. Some of the things that I had in mind was to see if there were any appropriate frameworks out there for making a simple REST API. I also wanted to find an ORM that was simple to use and mature enough that more complicated queries wouldn't be a problem. Simply because I am lazy, and don't want to write my queries in SQL.

The framework

Looking into the possible web frameworks for exposing a REST API I ended up looking primarily at Gin and Echo. Gin is a lightweight framework with focus on performance, and especially its HTTP router and error handling are intriguing. It also has a built in JSON validator. Echo, on the other hand, is even more minimalist than Gin, and while it does not have a built in JSON validator, they do recommend the same one used by Gin. The router is also based on radix tree, just like Gin's. What excites me the most about Echo though, is its built in ability to bind incoming request data to a struct, as well as the context-based request handling.
If you take a look at the documentation for each framework, you will find that the documentation for Gin is very sparse. They do have a large community around the framework, but that is not going to cut it for me. While Echo's documentation is far from as good as it can get, it is still a whole lot better that of Gin. Furthermore, from what I can read on various forums online, it seems like there is a little too much magic going on behind the curtains with Gin, which makes it really hard to understand what is going on under the hood. For all of these reasons I decided to give Echo a try.

The ORM

As I mentioned earlier I want to use an ORM for my database queries. There are a few different options out there, all of them with different approaches. The ones I looked into are SQLBoiler, GORM and ent.
Ent seems to be an ORM that focuses primarily on representing database schemas as graph structures, which is great if you want to expose an API with query languages such as GraphQL. This is not the approach I want to take. My focus will be on making a simple REST API for my application to use.
SQLBoiler is more interesting in my opinion. It takes a database-first approach, meaning that it generates code from the schemas in your database. This means that it can create optimised code that is specific to your database. However, this also means that each time you make a change to your database schemas, you have to regenerate the code. I do have a rough idea of the structure of the schemas, but I am in no way confident that they aren't going to change over time, and they might even change quite often.
GORM seems to me to be a much more traditional ORM, especially for someone coming from Django. It takes a code-first approach, meaning that you define your database schemas by defining them as structs. It seems to be relatively user friendly, and it has OK documentation and a large community behind it. It also allows you to write raw SQL, should the ORM fall short in some cases, and you can define column constraints with tags on the struct.

I ended up going with GORM, because it is easy to get started with the schema definition, and wanted something relatively close to what I already know. I also don't see myself needing to do any graph traversals in this project.
Working on the project, I have found the ORM to be a little difficult to work with, but it might also be a learning curve thing. I'll get more into this in a later post.

The database

Looking at most of my needs and requirements, I could probably choose any old RDBMS. Especially because GORM has drivers for most of them. I do however have one very specific requirement.
When I defined the requirements in my previous blog post, I told you that the user should be able to find events near them. Defining what is nearby means knowing the location of the user, knowing the location of a given event (Both of them located in latitude and longitude), and calculating the distance between the two points.
We could choose to just store the latitude and longitude of an event in the database, query all events that match the search criteria made by the user, make the distance calculations in Go, sort the events by distance and discard the ones that are too far away, or limit by an arbitrary number. To me this seems like a waste of resources, and I would much rather that the database could make the distance calculations and return a result that has already been filtered and ordered, ready to return to the user.

So how do I get a database to make these calculations? I looked into possible solutions and other people who had the same problem and came across PostGIS. PostGIS is an extension for PostgreSQL that allows you to store, index and query geospatial data. I looked a little further at the documentation, and it turns out that making the calculations directly in the query is relatively straight forward. BINGO!
Because of this, my choice naturally fell on using PostgreSQL as my RDBMS for the project.

Final thoughts

Looking into the feasibility of using Go for my REST API, we have explored different options for frameworks, ORMs and RDBMS's. I chose to go with Echo web framework, GORM and PostgreSQL. I have since I made my choices been completely submerged in learning these new technologies, and coding away. I had forgotten just how much fun it is to build something from scratch. The process has also made me realise just how complete a framework Django is. It really does come with batteries included!
In the coming blog posts I'm going to write about my initial thoughts of programming in Go, show you some of the code I've written so far, and talk about the challenges and intricacies of the technologies I've decided to work with. Strap in, it's going to be a hell of a ride!

The events application should be able to:

• Have users using SSO (Single Sign On)

• Create new public events:

  • Have an event title

  • A text description of the event in question

  • Include photos for banner as well as in the description (upload)

    • The photos in the description on top of the ability to have one as a banner opens up for customisability of the event page, which for now I am going to limit in the interest of time-saving. The event page should have text fields for headline and body.

  • Hold a time range for the event

  • Hold an address/geolocation

  • Have one or more owners of an event

  • Add tags to event

  • Make event private (Not shown on search)

• Interact with events:

  • Subscription to events (push-messages for event updates, cancellation, etc.)

  • Add attendance to event (Automatic subscription)

  • Add event to your phones calendar

    • Make change to calendar in case of changes to time or location or in case of cancellation (Push message with prompt to change)

  • Suggest changes to event if user is not the owner

  • Share the event with other people via link (Also the way to invite people to private events)

  • Show event on a map (potentially opened in google maps)

  • Add comments to events (Future editions)

• Find events:

  • Search for events

  • Filter events using tags

  • Filter events using time (Think "events today")

  • Filter events using geolocation

    • Adjust maximum distance

  • Show events near the user on list (using filters if any)

  • Show events near the user with pins on a map (using filters if any)

  • Favorite an event (With potential subscription)

Now for the sake of granulation, and nut cluttering the minimum viable product (MVP) I am going to cut out SSO and the features it enables, and save it for the next iteration of the app. This means that attendance and favorite events will not be in the MVP as well as event ownership. However, I still want to pick a framework for developing the api which have dedicated libraries for easy implementation of SSO.

System requirements

Now that we have looked into the user requirements, let's have a look into what this means for the requirements of the system. As I have previously stated in my initial blog post, I want to make a phone and web application that interacts with the backend through an API. Therefore I am going to split the requirements into server side requirements and client side requirements.

Server side requirements

So what do these user requirements tell us about the requirements of the server side application?
There are a few things we can reasonably deduct:

• The application is primarily a simple CRUD application, and there will not be large amounts of heavy computing involved

• There will be more people looking for events in their vicinity than people creating these events, which means that there is going to be considerably more reads to the database than writes

• Events will be filtered and found using geolocation, timestamps as well as tags. This means that we will have to use a data-model that can store and effectively search in such datatypes. Especially geospatial search could prove to be a bit tricky.

• The system should be able to store and easily look up high resolution images for rendering on the events

Client side requirements

The client should:

• Be able access the users geolocation

• Have easy access to use of maps - Preferably google maps, as this is what most users customarily use and thus has the interface they know the best.

• Be able to push messages to the user

• Have the ability to make changes to said event in users phone, if granted access

Performance requirements

Taking into consideration that this project is chiefly aimed at smartphone users, we can make some general requirements that boil down to user experience:

• It should not take more than two seconds from searching/filtering events until a result is displayed on the client application

When it comes down to the server side application, we can be relatively lax as this is a hobby project, which is more about learning new technologies and honing my skills, than it is about amassing a huge user base. However, if the project should for some reason become popular, it would be prudent to design the system with scalability in mind. For now, let's stick to some simple and not overly ambitious requirements:

• The system should be able to handle that api requests are replied to within two seconds, for up to 100 requests per second. Let's say that this should be the case for the 95th percentile of all requests.

• A maximum of 2000 simultaneous sessions at a time.

• Zero downtime is not necessary. For now, new features can be introduced at night time, although error handling should be handled with a type of middleware. Error handling and response to end user on failure is a whole beast on its own, which should definitely be touched upon later in the process.

Have I forgotten anything when writing down the requirements for the project? Definitely! For now we are going to head on to the next thing, but I will revisit the requirements in a later blog post or cheat a little and add the requirements in this post as they appear 😉

Choosing the technologies

It's time for the difficult part. Actually picking which programming languages and technologies to use! As I mentioned in my previous blog post I am looking to learn new languages and technologies, and maybe brushing up on things that I have previously worked with. But I have been working with the python framework Django the past three years, and the more I think about what the application should be capable of, the more I think that Django is a great choice for the backend part of the project. It has a great framework for quickly exposing an API called django REST framework, it has built in user management, a really well made ORM, and an API for modelling with geospatial data! It also has a great template language for the frontend part based on Jinja2 called Templates, if one wants to use it for that part as well. Did I mention that it comes with built in admin site that can be customised and used as a backoffice?
But no, damnit! I am only doing this project not just to show my capabilities and my thought process, but to learn something new as well. A language that I have been wanting to try out for a while is Golang. So does it even make sense to make the server side part using Go? In my next post we're going to have a look at how feasible this actually is. I'm not going to go into details about Go as a language, but rather if there are any frameworks and libraries that can help me build the application so I can prevent having to build everything from scratch, thus completely exploding the scope of this project.

I have long been pissed at facebook. Partly because it keeps people, including myself, in an infinite death scroll whenever they decide to open the app. But primarily my problem with facebook is that it still have a few features which no one have managed to do better, or at least have not been successful in poaching the users to their proprietary platform. This in turn forces people, such as myself, to stick with facebook even though they don't particularly enjoy using the platform. I am talking about groups and events. Now, I am aware that there are alternatives out there - groups.io and eventbrite are good examples. However, I want to start a project to challenge myself to learn languages and technologies I have either not used before, or just want to get more practice using, and I am bringing you along for the ride!

The concept

I want to build an application for events. This is not supposed to be an app for companies and people who want to manage their events, but rather an app for finding public events near the user and creating private events for friends and family.
The purpose is to make it as easy and simple as possible to find, filter and plan attendance to any event in your vicinity, or even plan ahead when travelling.

Choosing the technologies

Hey, that sounds really cool, Alexander! What technologies are you going to use for your application? I'm really glad you asked! The answer is that we're about to find out together. To make it easier to select the appropriate technologies, first we should look at some general requirements and scope of the project, which will be the extend of this article. Once we have a better understanding of the extend of the project we will define the business and user requirements. When we have these we can deduct what is required of the system, choose our technologies and create the appropriate system design. Let's get into it.

Scope

The project is limited by the fact that I am only one person working on the project, and the focus is to build something that is realistically finished. Therefore it is important to limit the features and complexity of the system. I still want this application to work across both iOS, android and as a web application. Therefore it could be beneficial to choose a cross-platform technology for the UI part of the application. If not, the application should be easily portable into other platforms. Realistically, most people would get the greatest benefit from using their smartphones for the functionality of the app, so this should be prioritised.

For server-side, I want to make an exposed api which can handle users, and create, read, update and delete events (CRUD), including geotagging, tags and other meta data about events. The server-side should be kept simple with as low complexity as possible. It should also be hosted on a suitable cloud service because I don't want to deal with the hassle and unreliability of having a server rack in my closet.

There are many frameworks out there intended for getting started easy with your api, all of which include varying amounts of of boilerplate code. In order to choose the right framework, we need to consider our needs and requirements, such as performance, built-in features and third party libraries, documentation and support. Keep in mind that, as I have mentioned, I am going to choose frameworks and technologies/languages that I have not used before or want more experience with.

UI

The UI technology used should either be easy to port to iOS and android, be cross-platform by nature, or be a hybrid web app, as most people would have most use of it on their phone (Think of finding an exhibition "near you"). I am not going to develop a native application for iOS and one for Android because of time and resource constraints.

The UI will be a simple search/filter page for finding events near the user, and should integrate a map for plotting the events, and making the overview better for the user.