Rootconf Hyderabad edition

On SRE, systems engineering and distributed systems

Tickets

Supporting utf8 characters in a utf8 mysql table

Submitted by Ritikesh (@ritikesh) on Wednesday, 14 August 2019

Section: Crisp talk (20 mins) Category: DevOps Status: Confirmed & Scheduled

View proposal in schedule

Abstract

Abstract:

MySQL’s “utf8” encoding only supports three bytes per character. The real UTF-8 encoding needs up to four bytes per character. After upgrading our app’s rails version in production, we started noticing issues in saving objects in the database because of MySQL’s strict mode. We’ll talk about key learnings, the migration itself and best practises for newer apps.

Target Audience:

Developers/Devops Engineers

Outline

  1. How we supported storing utf8 content in our app.
  2. How upgrading the rails version caused MySQL to run in strict mode causing the problems.
  3. How we evaluated our options and chose the final solution.
  4. The actual migration and dealing with ad-hoc issues.

Requirements

NA

Speaker bio

Lead Software Engineer @ Freshworks.

Links

Comments

  •   Anwesha Sarkar (@anweshaalt) Reviewer 6 months ago

    Hello Ritikesh,

    Thank you for your submission. Here is the feedback for your proposal:

    1. How much data you are talking about?
    2. Mention the names of the option.
    3. What is the final solution you went for?
    4. The abstarct and outline needs to be elaborate.

    Submit your slides and preview video by 23rd September, 2019, it helps us to give a fair review to the proposal.

    Cheers
    Anwesha

  •   Ritikesh (@ritikesh) Proposer 6 months ago (edited 6 months ago)

    Hi Anwesha,

    1. ~ 10 TB
    2. The options included:
      • Serialising selected fields at the ORM level(serialized fields in ActiveRecord) - No framework level changes required
      • Migrating entire DB/tables to utf8mb4 and running rails in utf8mb4 format
      • Hybrid approach - migrating only selected columns across tables and running rails in utf8mb4 format
    3. We went with the third option(hybrid) as it allowed us to go ahead with minimal changes and maximum flexibility

    I’ll update the abstract/outline with a link to a blog(on-going) post where I have described the problem statement, challenges and solutions in detail.

    Thanks,
    Ritikesh

  •   Ritikesh (@ritikesh) Proposer 6 months ago

    Hi Anwesha,

    I’ve updated the links section with a copy of the content of the blog to be published. Please let me know if you need anything else.

    Thanks,
    Ritikesh

  •   Ritikesh (@ritikesh) Proposer 6 months ago

    Hi Anwesha,

    Do you have an update on this or the other talk that I’ve submitted?

Login with Twitter or Google to leave a comment