Handling updates received via websocket. Creating something more

12.03.2019

The rules were simple. Each user could choose one of 16 colors and paint one pixel anywhere on the canvas with it. It was possible to paint as many pixels as you wanted and with whatever colors you wanted, but in order to recolor the next pixel, you had to wait 5 minutes.

True, the rules said: “By coordinating with others, you can create much more than by acting alone.”

What happened over the next 72 hours shocked the organizers. This appeared on an empty canvas:

Each pixel on the canvas was placed manually. Every icon, every flag, every meme was painstakingly created by hundreds of thousands of people who had nothing to do with each other except an internet connection. So, one way or another, but what happened on Reddit can rightly be considered the birth of art.

How it all happened

It is impossible to describe it in a few words. Countless dramas took place on the canvas - fights, battles and wars, sometimes it’s not even clear for what reason. They were conducted on small forums, in private chats, there were so many of them and they all happened at once, so it was not possible to keep track of everything. In general, the canvas was traced eternal history about the three forces necessary for humanity to create.

Creators

The creators came first. They were artists for whom the pure canvas has an irresistible attraction.

The creators began to recolor the pixels randomly, just to see what they could do. Therefore, the first drawings looked more like rock art– the artists were just beginning to spread their wings.

Pretty quickly, they realized that working alone and placing only one pixel every 5-10 minutes, it is impossible to create anything significant. Someone is bound to ruin their work. To create something more, they must work together.

And then someone suggested drawing on a grid that would clearly show where the next pixel needed to be painted in order to get a coherent image. So in the lower left part of the canvas appeared Dickbutt - a famous Internet meme, the fruit of a teenage sense of humor. It became the first joint work.

But the creators did not stop there. They began to add various elements to Dickbutt, paint it in different colors and even tried to transform it into Dickbutterfly. Behind this stupid idea was a hint of an impending creative tsunami.

However, this did not happen immediately. The creators were intoxicated by their power. Next to Dickbutt, a Pokemon Charmander appeared, in which, instead of a paw, a member began to grow, and then two more.

It was no longer a design. Some creators have tried desperately to remove the provocative additions, calling for "pure" art, but others have persisted. But it was not there.

It became clear that too much freedom leads to chaos. Creativity needs limits just as much as it needs freedom. When someone can put any pixel anywhere, how can that not lead to mayhem?

Guardians

This problem was very quickly solved by another type of users - keepers. They came with one goal - to conquer the whole world.

Having formed fractions by color, they began to conquer space by painting it in a certain color. One of the first and largest was the Blue Corner faction. Appearing in the lower right corner, it spread like a plague. Her followers proclaimed that in this way they should curl the entire space of the canvas. Pixel by pixel, they began to translate their idea into reality, capturing huge areas soon.

Blue Corner was not alone in its endeavors. On the other side of the canvas, another group appeared - Red Corner (red corner). Its participants said that they are adherents of the left political views. Another group - Green Lattice (green lattice) - took up the ubiquitous interspersing of green and white pixels. It proved to be highly efficient, as it had to paint half as many pixels as the other factions.

Keepers went to the creators in a frontal attack. Charmander became the first site of the battle. Upon discovering that the Blue Corner had begun slaughtering the Pokémon with blue pixels, the creators realized the threat and ended the internecine wars.

They fought back, replacing every blue pixel with their own. But the forces were not equal. Thanks to his determination, Blue Corner has gathered a much larger army than the creators. And the only thing left for the creators to do in such a situation was to beg for their lives.

And somehow it turned the tide. At the Blue Corner, a debate began about their role in creative process. One participant posed the question, “Since our wave inevitably takes over the world completely, should we show mercy to other art forms that we encounter?”

It was a question that sooner or later confronted every faction. For all their expansionist zeal, what were they to do with the art that stood in their way?

This was the turning point. Mindless factions turned into defenders.

But it wasn't over yet

In a world filled with predatory color, the creators were able to return to their creations. By adding one element after another, they began to make them more complex. Using three-pixel fonts, they began to write texts. One of the most famous creations was the Star Wars prequel.

The creators united in groups working on a common project. They shared strategies and patterns among themselves. One of the most successful was the group that created the Windows 95 panel with the Start button in the corner.

Others have created a block of hearts like in old video games like Zelda. Few started this project, but others quickly joined them, and as a result, hearts, painted in the colors of various flags, stretched to half the canvas.

Another group recreated Van Gogh's Starry Night.

However, not everything went smoothly. Defenders who once welcomed the creation of works of art have become fashion tyrants. They began to specify what can be created and what not. It began shortly before the creators began to create according to their own rules.

The factions turned their eyes on each other, demanding that their followers take sides in epic battles. They didn't have time to pay attention to the pitiful pleas of the creators who wanted to get approval for the ideas of the new art.

Fighting between the defenders flared up serious. Twitch live-streamers encouraged their followers to attack Blue Corner and Purple. Battle plans were being made. They called for emotions.

There have even been feint attacks, where adherents of the same color placed opponent pixels inside their own so that they can complain about the violation and attack back.

However, the biggest problem was a strict rule - the canvas cannot be enlarged. Both the warring factions and the creators began to realize that they simply would not have room for new art.

From the very beginning, flags appeared on the canvas various countries. They grew and bumped into each other. A real epic battle broke out between the flags of Germany and France. It became clear that an intermediary was needed to develop new spaces.

Suddenly, the world, having escaped primitive raids at the beginning, became ready for full-scale war. Desperate attempts to solve the problem through diplomacy came to nothing. Meeting in chats, the leaders of the creators and defenders only blamed each other.

We needed a slicker with whom everyone could agree.

Destroyers

On the Internet site 4chan drew attention to what was happening on Reddit. And they couldn't get past. Their users have chosen the color closest to their heart - black. They became the Void.

As a tear slowly spreads over the surface, so black pixels began to appear in the center of the canvas, destroying everything in its path.

At first, other factions tried to make an alliance with them, naively believing that diplomacy would work. But they failed because the Void was different.

The void was no protector. Unlike other factions, she did not show any loyalty to the arts. The followers of the Void practiced destructive egalitarianism under the slogan "The Void will swallow everything." They didn't make contact with others. They just wanted to paint the whole world black.

And that was exactly what was required. On the verge of extinction, all the members of the project banded together to fight the Void to save their art.

But the Void was not so easy to defeat, because it was needed. It was necessary to destroy everything so that a new art, the best, would be reborn from the ashes. And without the Void it was impossible.

So the Void became the catalyst for the creation of the largest work of art.

From the very beginning there was a stubborn struggle for the central part of the canvas. The creators claimed this territory for their works. At first they tried to do it with the help of icons. Then in a coordinated attempt to create a prism like on the cover Album Pink Floyd back side Moon."

But the Void ate everything. One after another, the creations created only warmed up her predatory appetite for chaos.

And yet, it was exactly what was needed. By destroying the art, The Void forced users to come up with something better. They knew they could defeat the black monster. They just need an idea with good potential that will attract enough followers.

And that idea was the American flag.

On the last day of the project, everyone came together to drive the void away once and for all. A coalition was created from people who would otherwise tear each other apart - from supporters and opponents of Trump, from Democrats and Republicans, from Americans and Europeans.

They teamed up to create something together, in this little corner of the internet, proving that in an era where such collaboration seems impossible, they can still do it.

The ancients were right

Shortly thereafter, the Reddit experiment ended. Today, he is accompanied by many stories told in dozens of chat rooms. Each work of art created in the project was covered by hundreds of new ones, of which only a few remained on the final canvas.

But the most surprising, perhaps, is that, despite the anonymity and lack of prohibitions, there were no racist or misanthropic symbols on the final canvas. It was a beautiful circuit of art, life and death. And he was not the first in our history.

Many millennia ago, when humanity (the real one, not just the one on Reddit) was still in its infancy, Hindu philosophers suggested that the heavens were made up of three competing, but necessary, deities: Brahma the Creator, Vishnu the Preserver, and Shiva- Destroyer.

Even without one of them, the universe would not be able to function. For there to be light, there must be darkness. For life to exist, death is needed. For creation and art there must be destruction.

Several days of the project showed that this approach proved to be prophetic. In the most incredible way, Reddit proved that creation requires the presence of all three components.

Final canvas

Facebook

Twitter

Pocket

LinkedIn

fb messenger

It cannot be said that 100% of corporate jokes on Humor Day are successful and engaging. This year, the Reddit administration launched Place, an interactive 1000 by 1000 pixel graphic canvas, and a section dedicated to it. It was assumed that the members of the community would jointly paint this canvas as they liked. But as a result, it turned into a battle for the Place, sometimes turning into a philosophical confrontation. An ordinary drawing exercise has turned into an exciting social experiment. The story from start to finish was documented by the Sudoscript blog.

The rules of the Place were simple. Each participant could choose one pixel from 16 colors and place it anywhere on the canvas. You could place as many pixels as you wanted, but you had to wait 5 minutes between each placement. After 72 hours these very simple rules led to the creation of an amazing collective canvas:

Each of the pixels visible above was placed manually. Every icon, every flag, every meme was painstakingly created by thousands of people who had nothing in common but an internet connection.

During creation, there were countless dramas, ideas, fights, even wars. But in general, the history of the Place is an eternal drama about the three forces that humanity needs to create and create and develop technologies.

Creators

First there were the creators. These were artists for whom the empty canvas seemed like an irresistible opportunity. Early artists placed pixels randomly, just to see what could be done. In the first minutes, the first sketches appeared. Rough and immature, they resembled the cave paintings of cavemen.

The creators immediately saw what power and potential the pixels hide. But working alone, they could place one pixel every 5 or 10 minutes. Creation meaningful drawing would take forever. To draw something, they had to work together.

Then someone came up with the brilliant idea of ​​using a grid for drawing, which would overlay the drawing and show where the next pixels should be. The first to go through this experiment was the well-known meme of the English-speaking Internet Dickbutt. And the inhabitants of the Place set to work: Dickbutt materialized in just minutes in the lower left corner of the canvas. The first creation of collective creativity appeared at the Place.

Then, when the creators got a little drunk on the possibilities, the Pokemon Charmander appeared, in which a member rather soon appeared instead of a leg. And the first conflict began: some creators diligently tried to clean up offensive drawings, but others persistently added obscene things.

The creators are faced with a fundamental philosophical problem: too much freedom leads to chaos. Creativity needs a limiter just as it needs freedom.

Defenders

The Place had a different type of user who had to deal with exactly this problem. But they started with more primitive goals: the conquest of the world. Dividing into factions by color, they tried to capture the Place. One of the first was the Blue Corner. It originated in the lower right corner and spread like a plague.

Another group founded the Red Corner on the opposite side of the canvas, they leaned towards the political left. Another group called the Green Grid painted the canvas through the pixel - green cells interspersed with white. Since they only had to paint half the pixels, they were more efficient than the other factions.

It wasn't long before the factions clashed with the creators. Charmander became one of the first objects of the battle. The blue corner began painting the Pokémon with blue pixels, and the Creators switched from "phallic wars" (who draws more members) to a more serious threat. They took the fight, painting every blue pixel with their own. But the quantitative advantage was not in their favor.

So the Creators surrendered to the mercy of the winner, and somehow it hurt the feelings of the Blues. Among them appeared doubting their role in the world of the Place. “Our wave will inevitably cover the whole world, from edge to edge, should we show mercy to other art that we encounter,” asked one of the group.

Each of the factions faced this issue. And everyone decided to save other drawings. So the colored waves began to flow around the drawings without painting over them.

This was the turning point. Mindless colored factions have become useful Defenders.

But it's not a happy ending

Finally, the insatiable color waves were stopped and the Creators could return to creativity. The drawings became more and more difficult. Texts written in pixels appeared.

The creators got together in small groups, creating subsections on Reddit where they could discuss draft drawings and strategy. One of the most successful groups drew the taskbar in the style of Windows 95. Another sketched Place with hearts.

Then came Van Gogh.

But everything was not so simple. The defenders turned into tyrants, dictating the style of the drawings. They decide what can be drawn and what is not. Factions began to divide users among themselves, calling for parties, meanwhile the Creators were waiting for the approval of new ideas.

The battles between the Defenders were getting tougher. One Twitch streamer called on his followers to attack the BLUs. Battle strategies were developed. There were even provocations: fans of the same color themselves drew pixels of the enemy's color on their territory in order to have an excuse for a retaliatory attack. While the factions fought among themselves, the Creators found that there was no room for new drawings.

The flags began to rise different countries As they grow, they inevitably run into each other. For example, the flags of Germany and France clashed on "no man's" territory.

The world seemed to be on the brink of war. All parties tried to resolve the conflict through diplomacy. The leaders of the Creators and Defenders chatted, but usually ended in mutual accusations.

The place needed a villain that everyone else could unite against.

Destroyers

The void has arrived.

It started with 4chan, the most famous image board in the world. The pranksters inhabiting it noticed what was happening on Reddit and could not pass by. They became the Void.

A spot of black pixels began to grow in the center of the Place. At first, the factions tried to make a pact with the Void through diplomacy. But they failed, the Void acted differently. She was not one of the Defenders, she was not guarding art. Her followers preached that the Void would devour everything. They didn't form sides, they just wanted to paint the whole world black.

It was exactly the kick in the ass that the Place lacked. Faced with a common threat, the Creators and Defenders once again united to save art. But the meaning of the Void was not just destruction, somehow it gave rise to a new, better art.

For example, the location in the center was one of the most contested among the creators. And when it turned black, the Defenders realized that they would have to come up with a better idea that would involve enough followers to fight the black monster. One of these ideas was the US flag.

On the last day of the Place, the most incredible coalition formed to fight the Void—Trump fans and Trump detractors, Republicans and Democrats, Americans and Europeans.

Soon the Reddit experiment ended. There was not a single racist drawing on the final canvas, not a single symbol of hatred.

Twitter

Pocket

LinkedIn

fb messenger

Reddit opened a self-expression page in honor of April Fools, but the project, created as a joke, has become a kind of wall for collective graffiti from participants from around the world, showing what mark they want to leave in history.

On April 1, Reddit launched the Place project, a page with a blank canvas on which each forum user could draw any image. The artists had limitations: they could only draw one pixel of one of the 16 colors every five minutes, and the size of the canvas was also limited. On top of the drawn pixels, you can draw others (and then the first author can again draw his own pixel on top of the opponent's pixel), which is why the authors of the images a priori conflict. As indicated in the description of the canvas, the project is designed for joint creativity - “Each of you can create something individual. Together you can create something more.”

At the start of the project, everyone literally pounced on the canvas - every pixel on it was filled. At first, users simply poked color into free pixels, but then teams began to form, which began to draw one of the simplest possible ideas that finds like-minded people - state flags. The more users worked on the canvas, the more interesting and complex ideas came to their minds. And "The Place" has evolved from an April Fool's project to a place where users from all over the world come together in real communities to show something to the world and support their creation from groups of raiders who really just also want to draw some kind of their own drawing.

Place during the first hours of operation

"The Place" has become an online sticker board, but each of them is created with great difficulty. For example, to draw a 48 x 68 pixel Linux logo and protect it from squatters requires 3,264 people to draw at the same time.

Users unite to implement ideas that are smaller and simpler, but still very team-oriented, like this row of hearts with flags of different countries (and not only): everyone is responsible for “their” heart, but all together people unwittingly form one team.

And others write whole canvases of text, for example, like this group of fans " Star Wars", who wrote and maintains Supreme Chancellor Palpatine's famous monologue about the Sith Darth Plagueis from the third episode of the space saga.

However, some users complained that the project participants "planted" bots instead of themselves, which automatically update the pixel occupied by the author every five minutes. Despite the fact that when drawing each pixel, the user needs to repeat a set of actions (for example, choosing a certain color), some have been able to bypass the protection mechanism and create bots that draw pictures for them.

However, there are those who still create special threads, trying to call on like-minded people who will help draw and leave for future generations something like ... puke Rick Sanchez from the Rick and Morty cartoon. Seriously? Are they really not bots?

After 72 hours of work, the project was closed. The administration of the resource thanked everyone for their participation and for the fact that people united "to create something more."

Reddit often becomes a place for various social activities. For example, recently sad users of the resource decided to ask what it's like to wake up every day with a smile. And before that, Reddit readers shared stories about girls with each other. Not only anonymous people are sitting on the popular resource: recently the actor answered questions from users about the film “Trainspotting” for several hours.

To begin with, it was extremely important to determine the requirements for the April Fools project, because it had to be launched without “overclocking” so that all Reddit users would immediately have access to it. If it had not worked perfectly from the very beginning, it would hardly have attracted the attention of a large number of people.

    The "board" needs to be 1000x1000 tiles in order to look very large.

    All clients must be in sync and display the same board state. After all, if different users have different versions, it will be difficult for them to interact.

    You need to support at least 100,000 users at the same time.

    Users can place one tile every five minutes. Therefore, it is necessary to maintain an average update rate of 100,000 tiles per five minutes (333 updates per second).

    The project should not negatively affect the work of other parts and functions of the site (even if there is high traffic on r/Place).

  • In case of unexpected bottlenecks or failures, flexible configuration must be provided. That is, you need to be able to adjust the size of the board and the allowed drawing frequency on the fly if the amount of data is too large or the refresh rate is too high.

Backend

Implementation decisions

The main difficulty in creating the backend was to synchronize the display of the state of the board for all clients. It was decided to have clients listen for tile placement events in real time and immediately request the state of the entire board. It is acceptable to have a slightly outdated full state if you subscribe to updates before the full state was generated. When the client receives the full state, it displays all the tiles it received while waiting; all subsequent tiles must be displayed on the board as soon as they are received.


For this scheme to work, the request full state boards should run as fast as possible. At first, we wanted to store the entire board on one line in Cassandra, and have each request just read that line. The format for each column in this row was:


(x, y): ('timestamp': epochms, 'author': user_name, 'color': color)

But since the board contains a million tiles, we needed to read a million columns. On our production cluster, this took up to 30 seconds, which was unacceptable and could lead to excessive load on Cassandra.


Then we decided to store the entire board in Redis. We took a bit field of a million four-bit numbers, each of which could encode a four-bit color, and the x and y coordinates were determined by the offset (offset = x + 1000y) in the bit field. To get the full state of the board, it was necessary to read the entire bit field.


Tiles could be updated by updating values ​​at specific offsets (no need to block or do the whole read/update/write procedure). But all the details still need to be stored in Cassandra so that users can find out who posted each of the tiles and when. We also planned to use Cassandra to restore the board when Redis crashed. Reading the entire board from it took less than 100 ms, which was quite fast.


Here's how we stored colors in Redis using a 2x2 board as an example:



We were worried that we might run into read throughput on Redis. If many clients were connecting or updating at the same time, then all of them simultaneously sent requests to get the full state of the board. Since the board was a shared global state, the obvious solution was to use caching. We decided to cache at the CDN level (Fastly), because it was easier to implement, and the cache was closest to the clients, which reduced the response time.


Full board state requests were cached by Fastly with a timeout per second. To prevent a large number of requests when the timeout expires, we used the stale-while-revalidate header. Fastly supports about 33 POPs that cache independently, so we expected to receive up to 33 full board state requests per second.


To publish updates to all clients, we used our websocket service. We have previously used it successfully to power Reddit.Live with over 100,000 concurrent users for live private message notifications and other features. The service was also cornerstone our past April Fools' projects - The Button and Robin. In the case of r/Place, clients supported websocket connections to receive real-time updates on tile placements.

API

Getting the full state of the board


At first requests got to Fastly. If it had a valid copy of the board, then it immediately returned it without contacting the Reddit application servers. If not, or the copy was too old, then the Reddit application read the full board from Redis and returned it to Fastly to be cached and returned to the client.




Please note that the request rate never reached 33 per second, i.e. caching with Fastly was very effective tool protecting the Reddit app from most requests.



And when requests did reach the application, Redis responded very quickly.

Tile drawing


Stages of drawing a tile:

  1. The timestamp of the last tile placed by the user is read from Cassandra. If it was less than five minutes ago, then we do nothing and an error is returned to the user.
  2. Tile details are written to Redis and Cassandra.
  3. The current time is recorded in Cassandra as the last time the user placed a tile.
  4. The websocket service sends a message about the new tile to all connected clients.

In order to maintain strict consistency, all writes and reads in Cassandra were performed using the consistent level QUORUM .


In fact, we had a race here whereby users could place multiple tiles at once. There was no blocking in stages 1-3, so simultaneous attempts to draw tiles could pass the test in the first stage and be drawn in the second. It seems that some users found this bug (or they used bots that ignored the limit on the frequency of sending requests) - and as a result, about 15,000 tiles were placed using it (~0.09% of the total).


Request rate and response time as measured by the Reddit app:



The peak tile placement rate was almost 200 per second. This is below our calculated limit of 333 tiles/s (average assuming 100,000 users place their tiles every five minutes).


Getting details on a specific tile


When requesting specific tiles, data was read directly from Cassandra.


Request rate and response time as measured by the Reddit app:



This request proved to be very popular. In addition to regular client requests, people have written scripts to retrieve the entire board one tile at a time. Since this request was not cached in the CDN, all requests were served by the Reddit application.



The response time to these requests was quite short and kept at the same level throughout the life of the project.

Websockets

We don't have separate metrics showing how r/Place has affected the performance of the websocket service. But we can estimate the values ​​by comparing the data before the start of the project and after its completion.


Total number of connections to the websocket service:



The base load before the launch of r/Place was about 20,000 connections, the peak was 100,000 connections. So at the peak we probably had about 80,000 users connected to r/Place at the same time.


Throughput of the websocket service:



At peak load on r/Place, the websocket service was transmitting over 4 Gbps (150 Mbps per instance, 24 instances in total).

Frontend: web and mobile clients

In the process of creating the front-end for Place, we had to solve many complex tasks related to cross-platform development. We wanted the project to work the same on all major platforms, including desktop PCs and mobile devices on iOS and Android.


The user interface had to perform three important functions:

  1. Display board status in real time.
  2. Allow users to interact with the board.
  3. Work on all platforms, including mobile applications.

The main object of the interface was the canvas, and the Canvas API was perfect for it. We have used the element 1000x1000 in size, and each tile was drawn as a single pixel.

Canvas drawing

The canvas had to reflect the state of the board in real time. It was necessary to draw the entire board when the page loaded and finish drawing updates coming through web sockets. A canvas element that uses the CanvasRenderingContext2D interface can be updated in three ways:

  1. Draw an existing image on the canvas with drawImage() .
  2. Draw forms using different form drawing methods. For example, fillRect() fills a rectangle with some color.
  3. Construct an ImageData object and draw it on the canvas with putImageData() .

The first option did not suit us, because we did not have a board in the form of a finished image. There were options 2 and 3. The easiest way was to update individual tiles using fillRect() : when an update comes through the websocket, just draw a 1x1 rectangle at position (x, y). In general, the method worked, but was not very convenient for drawing initial state boards. The putImageData() method was much better: we could determine the color of each pixel in a single ImageData object and draw the entire canvas at once.

Drawing the initial state of the board

Using putImageData() requires defining the state of the board as a Uint8ClampedArray , where each value is an eight-bit unsigned number between 0 and 255. Each value represents some color channel (red, green, blue, alpha), and each pixel needs four element in the array. A 2x2 canvas requires a 16-byte array where the first four bytes represent the top left pixel of the canvas and the last four bytes represent the bottom right.


Here's how the canvas pixels are associated with their Uint8ClampedArray representations:



For the canvas of our project, we needed an array of four million bytes - 4 MB.


In the backend, the state of the board is stored as a four-bit bit field. Each color is represented by a number between 0 and 15, which allowed us to pack two pixels into each byte. To use this on a client device, you need to do three things:

  1. Pass binary data from our API to the client.
  2. Unpack data.
  3. Convert 4-bit colors to 32-bit.

To transfer binary data, we used the Fetch API in those browsers that support it. And in those that do not support, used XMLHttpRequest with responseType set to "arraybuffer" .


The binary data received from the API contains two pixels in each byte. The smallest TypedArray constructor we had allows us to work with binary data in the form of one-byte units. But they are awkward to use on client devices, so we unpacked the data to make it easier to work with. The process is simple: we iterate over the packed data, pull out the high and low bits, and then copy them into separate bytes in another array.


Finally, four-bit colors had to be converted to 32-bit.



The ImageData structure we needed to use putImageData() requires that final result was in the form of Uint8ClampedArray with bytes encoding color channels in RGBA order. This means that we had to do one more unpacking, splitting each color into component channel bytes and putting them in the correct index. It's not very convenient to do four writes per pixel. Fortunately, there was another option.


TypedArray objects are essentially array representations of ArrayBuffer. There is one caveat here: multiple TypedArray instances can read from and write to the same ArrayBuffer instance. Instead of writing four values in an eight-bit array, we can write one value to a 32-bit one! By using a Uint32Array to write, we were able to easily update tile colors by simply updating one index of the array. True, we had to save our color palette in byte-reversed-byte order (ABGR) so that the bytes automatically fall into the correct places when read using Uint8ClampedArray .


Handling updates received via websocket

The drawRect() method was well-suited for drawing pixel-by-pixel updates as they were received, but there was one weakness: Large chunks of updates coming at the same time could lead to stuttering in browsers. And we understood that updates to the state of the board can come very often, so the problem had to be solved somehow.


Instead of immediately re-rendering the canvas every time we receive an update via websocket, we decided to make it so that websocket updates that arrive at the same time can be bundled and rendered in bulk at once. To achieve this, two changes were made:

  1. Stop using drawRect() - we found convenient way update many pixels at a time with putImageData() .
  2. Transferring canvas rendering to the requestAnimationFrame loop.

By wrapping the render in the animation loop, we were able to immediately write websocket updates to the ArrayBuffer while deferring the actual rendering. All websocket updates arriving between frames (about 16 ms) were bundled and rendered at the same time. Thanks to the use of requestAnimationFrame , if the rendering took too long (longer than 16ms), it would only affect the refresh rate of the canvas (rather than degrade the performance of the entire browser).

Interaction with canvas

It is important to note that the canvas was needed in order to make it more convenient for users to interact with the system. The main interaction scenario is the placement of tiles on the canvas.


But rendering each pixel accurately at a 1:1 scale would be extremely difficult, and we would not avoid mistakes. So we needed zoom (big!). In addition, users needed to be able to easily navigate the canvas, as it was too large for most screens (especially when using zoom).

Zoom

Since users could place tiles once every five minutes, placement errors would be especially frustrating for them. It was necessary to implement a zoom of such a multiplicity that the tile would be large enough, and it could be easily placed in Right place. This was especially important on touchscreen devices.


We implemented a 40x zoom, that is, each tile had a size of 40x40. We wrapped the element V

, which has CSS transform: scale(40, 40) applied to it. This was a great solution for tile placement, but made it difficult to see the board (especially on small screens), so we zoomed in two stops: 40x for drawing tiles, 4x for viewing the board.


Using CSS to scale the canvas made it easy to separate the code responsible for drawing the board from the code responsible for scaling. But this approach has several drawbacks. When scaling an image (canvas), browsers by default apply image smoothing algorithms. In some cases, this does not cause inconvenience, but it simply destroys pixel graphics, turning it into soapy porridge. The good news is that there is an image-rendering CSS property that allows us to "ask" browsers not to apply anti-aliasing. The bad news is that not all browsers have full support for this property.


Zoom blur:



For such browsers, another way of scaling had to be found. I mentioned above that there are three ways to draw on canvas. The first one, drawImage() , supports drawing an existing image or another canvas. It also supports image scaling during rendering (increasing or decreasing). While zooming has the same blurring issues as the above CSS, they can be solved in a more generic way in terms of browser support by clearing the CanvasRenderingContext2D.imageSmoothingEnabled flag.


So we've solved the blurry canvas problem by adding another step to the rendering process. To do this, we made one more element , which is the same size and position as the container element (that is, the visible area of ​​the board). After redrawing the canvas using drawImage() , the visible part of it is drawn in the new canvas at the desired scale. Because this extra step slightly increases rendering costs, we've only used it in browsers that don't support the image-rendering CSS property.

Navigating the canvas

Canvas is pretty big image, especially when zoomed in, so we needed to be able to navigate through it. To adjust the position of the canvas on the screen, we used the same approach as in the case of scaling: we wrapped the element to another

, which has CSS transform: translate(x, y) applied to it. Thanks to a separate div, we were able to easily control the order in which the transformations were applied to the canvas, which was necessary to prevent the "camera" from moving when changing the zoom.


As a result, we have provided support different ways camera position settings:

  • "Click and drag" (click-and-drag, or touch-to-drag);
  • "Click to move" (click-to-move);
  • keyboard navigation.

Each of these methods is implemented differently.

"Click and drag"

This is the primary way to navigate. We stored the x and y coordinates of the mousedown event. For each of these events, we found the offset of the mouse cursor position relative to the initial position, and then added this offset to the existing canvas offset. The camera position was updated immediately, so the navigation was very responsive.

"Press to move"

When you click on a tile, it is placed in the center of the screen. To implement this mechanism, we had to keep track of the distance between the mousedown and mouseup events in order to separate "clicks" from "moves". If the distance the mouse moved was not enough to be considered "moving", the "camera" position was changed based on the difference between the mouse position and the point in the center of the screen. Unlike the previous navigation method, the "camera" position was updated using the easing function. Instead of immediately setting a new position, we saved it as a "target". Inside the animation loop (the same one used to redraw the canvas), the current “camera” position was moved closer to the target using the easing function. This made it possible to get rid of the effect of too abrupt movement.

Keyboard navigation

It was possible to navigate the canvas using the keyboard arrows or WASD. These keys controlled the internal motion vector. If none of the keys were pressed, then the default vector had coordinates (0, 0). Pressing any of the navigation keys added 1 to x or y. For example, if you press "right" and "up", then the coordinates of the vector will be (1, -1). This vector was then used inside the animation loop to move the "camera".


During the animation, the movement speed was calculated depending on the zoom level using the following formula:


movementSpeed ​​= maxZoom / currentZoom * speedMultiplier

When the zoom was disabled, the buttons were faster and much more natural.


Then the motion vector was normalized, multiplied by the motion speed and applied to the current position of the "camera". Normalization was used to match the speed of diagonal and orthogonal movements. Finally, we applied the easing function to changes in the motion vector itself. This smoothed out changes in movement direction and speed, so that the "camera" moved much smoother.

Mobile Application Support

When embedding canvas in iOS and Android applications, we encountered some difficulties. First, we needed to authenticate the user so they could place tiles. Unlike the web version, where authentication is session-based, in mobile applications we used OAuth: in this case, applications should provide the logged in user with a WebView with an access token. The most secure way to implement this is by injecting OAuth authorization headers via a JS call from the application to the WebView. This would allow us to customize other headers if necessary. Then it was just a matter of parsing the authorization headers on every API call:


r.place.injectHeaders(('Authorization': 'Bearer ’});

In the iOS version, we additionally implemented support for notifications when a user's tile was ready to be placed on the canvas. Since the placement was done entirely in the WebView, we had to implement a native application callback. Luckily, in iOS 8 and up, this is done with a simple JS call:


webkit.messageHandlers.tilePlacedHandler.postMessage(this.cooldown / 1000);

The delegate method in the application then dispatched notifications based on the recharge timer passed to it.


What have we learned

Always missing something

We planned everything perfectly. We knew when the launch would be. Everything had to go like clockwork. We had front-end and back-end tested under load. We humans just couldn't make any more mistakes. Right?


The launch went really smoothly. During the morning, as r/Place grew in popularity, the number of connections increased and traffic to WebSocket instances increased:




We foresaw it. And we were preparing for the fact that as a result the network would become a bottleneck in our system. But it turned out that we have a large stock. However, looking at CPU usage, we saw a very different picture:



These are eight-core machines, so it was obvious that they had reached their limit. Why did these "boxes" behave so unexpectedly? We decided that the load generated by Place is very different in nature from what it was before. In addition, a large number of very small messages were used, while we usually send larger messages like live thread updates and notifications. Also, we usually don't have that many users getting the same message. So the working conditions were very different from the usual.


We decided that nothing terrible is happening: we scale up and that's it. The responsible employee simply doubled the number of instances and went to the doctor without a gram of excitement.


And then this happened:



At first glance, nothing special. If not for the fact that this was our RabbitMQ production instance, which handles not only websocket messages, but everything that reddit.com depends on. And it wasn't good. Not good at all.


After numerous investigations, hand-wringing, and instance upgrades, we narrowed down the source of the problem to the management interface. It always seemed somehow slow, and we decided that it was regularly requested by our Rabbit Diamond collector . We thought that the additional communication associated with the launch of new websocket instances, combined with the mass of messages received in connection with this exchange, led to an overload of the Rabbit, which was trying to keep track of the execution of requests to the admin panel. So we just turned it off - and the situation improved.


But we do not like to be in the dark, so on hastily bungled a makeshift monitoring script:


$ cat s****y_diamond.sh #!/bin/bash /usr/sbin/rabbitmqctl list_queues | /usr/bin/awk "$2~//(print "servers.foo.bar.rabbit.rabbitmq.queues." $1 ".messages " $2 " " systime())" | /bin/grep -v "amq.gen" | /bin/nc 10.1.2.3 2013

If you're wondering why we kept adjusting pixel placement timeouts, the answer is that we were trying to reduce the load on the entire project. For the same reason, for some time, some pixels were not displayed on the board for a long time.


Unfortunately, despite such messages:



The cooldown changes mentioned here were purely technical reasons. Although after them it was interesting to watch the r/place/new branch:



Perhaps this was part of the users' motivation.

Bots will stay bots

At the final stage of the project, we encountered another turmoil. We regularly have problems with customers misbehaving in terms of retry attempts. Many clients encounter errors and simply resubmit requests. And again. And again. That is, when some problem appears on the site, this leads to a wave of repeated requests from customers who do not know what excerpt is.


When we turned off Place, the endpoints that were accessed by a lot of bots started returning non-200th errors. This code was not very successful. Fortunately, all these repeated calls were easily blocked at the Fastly level.

Creating something more

r/Place wouldn't be so successful if it wasn't for the well-coordinated teamwork. We'd like to thank u/gooeyblob, u/egonkasper, u/eggplanticarus, u/spladug, u/thephilthe, u/d3fect and everyone else who helped make this April Fool's experiment a reality.



Similar articles