Data Architecture

Data is the glue that holds distributed systems together. Data has it's own lifecycle starting from origination to flow, transformation, modification and finally deletion. At a high level these are some of the data design considerations



Database Type 

  • Relational, KV-store, Column Family, Document, Graph etc


Indexing and Querying

  • LSM, B-Tree, B+-Tree
  • R/W Access patterns
  • Range queries


Schema, Metadata, Encoding and Evolution

  • Versioning
    • Forward and Backward compatibility
  • Format
    • Avro, Thrift, Protobuf


Transaction

  • Atomicity
  • Isolation level
    • Lost update
    • Read committed
    • Read repeatable (snapshot isolation - Readers never block writers and vice versa)
    • Prevent lost update (read-modify-update cycle)
      • CSET, Atomic write, Conflict resolution, 
    • Phantom (generalisation of lost update)
      • Materialising conflict
    • Serialisability
      • Actual serial execution
      • 2 phase locking
        • Writers don’t just block other writers; they also block readers and vice versa. 
        • Deadlocks
        • Predicate lock, index range lock
      • SSI (Optimistic concurrency control)
        • all reads within a transaction are made from a consistent snapshot of the database
        • Pessimistic concurrency control - 2PL, ASE
        • Optimistic concurrency control - MVCC
  • Data durability 
    • WAL Logs
    • Backup and Restore
    • Error handling, RRD, Save points


Replication

  • Master slave
    • Read your writes, monotonic read, consistent prefix reads
  • Multi master (couchdb)
    • Conflict resolution / avoidance
  • Leaderless (Dynamo style)
    • Quorum, Sloppy quorum, LWW, Concurrent writes, Version vectors


Data partitioning

  • Hotspot
  • Partition by Key range, consistent hashing, Compound primary key
  • Partition index - Local index (scatter-gather), Global index
  • Rebalancing - Fixed, Dynamic
  • Request routing 


Trouble with Distributed systems

  • Faults and partial failures
  • Unreliable networks - synchronous vs asynchronous networks
  • Unreliable clocks
    • Don’t rely on the accuracy of the clock
    • NTP sync is not accurate
    • Confidence interval in time - Spanner
  • Process pause
  • Knowledge truth lies
    • Truth is defined by majority
    • Fencing tokens is required for distributed lock
    • Safety and liveness


Linearisablity

  • Serializability is an isolation property of transactions
  • Linearisablity is a recency guarantee
  • To make a system appear as if there is only a single copy of the data. 
  • Single leader replication is potentially linearisable. Multi-leader and leaderless are not linearisable
  • A network interruption forces a choice between linearisability and availability. 
  • CAP - Linearisable / Available when partitioned
  • The reason for dropping linearisability is performance , not fault tolerance.
  • Most DB are neither CAP-linearisable nor CAP-available 
    • MVCC is intentionally non-linearisable
    • Single leader replication (async) is non-linearisable
    • Partition makes it non-available
  • ZooKeeper by default is neither CAP-consistent (CP) nor CAP-available (AP) – it’s really just “P”. 
    • You can optionally make it CP by calling sync if you want, 
    • and for reads (but not for writes) it’s actually AP, if you turn on the right option.


Error handling , Recovery

  • Replay logs
  • Discover partitions
  • Replication catchup
  • Restores


Source of truth 

  • Primary DB
  • Derived data
    • Caches
    • Materialised Views
    • CQRS


Batch processing

  • Putting the computation near the data
  • Mapping and Chaining
  • Sort merge joins
  • GroupBy
  • Handling Skew
  • MapSide joins
    • Broadcast hash joins - joining a large dataset with a small dataset
    • Partitioned hash joins - partition and reduce the dataset
    • Mapside merge joins - if input is partitioned and sorted appropriately
  • Use cases - Search index, Key Value stores
  • The output of a reduce-side join is partitioned and sorted by the join key, whereas the output of a map-side join is partitioned and sorted in the same way as the large input 
  • Input is immutable and no side effects
  • In an environment where tasks are not so often terminated, the design decisions of MapReduce make less sense.
  • Materialising intermediate state
    • Makes fault tolerance easy
    • Sometimes they are not needed


Streaming processing

  • Messaging
    • Brokers
      • Routing: Fanout, load balancing
      • Acknowledgement and redelivery
      • Message ordering 
  • Partitioned logs - Kafka
    • Unified log
    • Partitioning to scale
    • Replayable
    • Consumer offsets
    • Durability
    • Immutable input and idempotent operations
  • CDC - Change Data Capture
    • Implemented using log based broker
    • Connectors - Debezium etc
  • Event sourcing
    • Immutable events written to event log (this is used in accounting where delta is captured)
    • CQRS - separate forms for read and write and allowing several different read views
  • Stream processing
    • Produce a new stream, real time dashboard, write it to a end database
    • Has operators - sorting doesn’t make sense
    • Uses: CEP, Stream analytics, Maintaining materialised views
    • Event time vs Processing time
    • Stream Joins
      • Stream-Stream joins (click through rate)
      • Stream-Table join (enrichment)
        • Similar to map side hash join
        • Table can be kept upto date using CDC
      • Table-Table join (materialised view maintenance)
        • Twitter
      • Time dependence of joins
        • Slowly changing dimension


Big Data

  • Data Analytics
  • ETL
  • Data warehouse 

Data Security

  • Encryption at Rest
  • Data Retention
  • Data classification
  • (Customer) Data isolation
  • Data Integrity
  • Data Availability
  • Data anonymisation
  • Backup and recovery
  • Database Security
  • Access Control




Overall Security

 

Microservices

Micro services though useful  come with a lot of baggage. Discretion is needed to decide if they're really needed. Monoliths are not bad. Most likely what is needed is a clean interface separation between various components in the monolith. 

Monoliths can serve for a long period unless you hit issues.
  • Release velocity is affected because of dependencies between components. This hampers development, testing and deployment time
  • Scaling characteristics of different components are different such that they cause unreliable use of the resources of underlying hardware due to differing traffic patterns
    • Capacity planning becomes hard
    • Performance becomes unpredictable
    • Resource exhaustion happens frequently and randomly
  • Need to develop and scale the component independently and make it available as a service

Micro-services takes a heavy toll on SRE; without the required automation and SRE firepower, it is really hard to maintain sanity of the entire system. With micro services proliferation the problem increases manifold with the web of inter service traffic.

Micro services needs to be implemented with discretion. Keep in mind the following considerations

Architecture        




12 factor app

https://12factor.net


Availability

How is the service fault tolerant?


Scalability

What’s the horizontal and vertical scalability?


Statelessness

Is the service stateless?


Async

Can it use Lambda / Async services?


Security Considerations

2FA, HTTPS, Tokens, Encryption, GDPR, Penetration testing, App testing


API

Contracts, Versioning, Dependency


Network

• Proxy

• Sync, Async, Batch 

• Multithreaded, Event based, Coroutine


Load Handling

• Load balancer

• Circuit breaker

• Throttling


Replication

Consistency


Data

• Transactions across services 

• Partitioning

• Schema, Metadata, Evolution

• Indexing, Querying

• DB type


Caching

• Object caching

• Page Caching


Service Mesh

• Istio


Shutdown

Graceful shutdown


i18n Considerations


SRE




Backup / Restore

• RPO - Recovery Point Object, 

• RTO - Recovery Time Objective


Reliability

• MTTF - Mean time to failure

• MTTR - Mean time to Recovery

• MTBF - Meantime between failure

• Uptime

• Fault tolerance


Performance / SLAs

• SLO's - Service Level Objectives

• Response time

• Latency

• Throughput

• Uptime


Release Management

Change Management

Config Management

• Zero Downtime upgrade, 

• Rolling deployments, 

• Automated deployments


Container and Orchestration

Docker / Docker Swarm or K8S


Dev / QA environment

Automated Dev / QA environments


CI/CD pipeline

Code Deploy, Circle CI, Codeship, Jenkins


Upgrades / (0 Downtime)

Zero downtime upgrade, Rolling upgrades, Canary rollout


Deployment

Ansible / Puppet


CI/CD pipeline

Code Deploy, Circle CI, Codeship 


Service Monitoring & Alerting

Pingdom, Nagios, CloudWatch, Prometheus, DataDog


Logging

Logstash, Fluentd


Cost

Cost tags, Analytics, Cost structure, Reserved Instances, Projections, Cost Optimisations

(Tools like Botmetrics)


Capacity Planning



Security

IAM Roles, Encryption, HTTPS


Networking

Diagram, VPC


Fleet management

Tagging, AMI images, Versions, Upgrades, Consolidation, Pruning


Incident Management and 

Incident Response

Outages, Load Management, Latency, Security Incidents


Process Management

Process group, Process monitoring 


OnCall

Pager Duty, VictorOps


Versioning and Packaging


Dev Process




Git Flow

Branching and Development process


API Docs

Swagger


Sentry

Error monitoring


Metrics

Concurrency, System metrics, Engineering Metrics


Testing

• Automation, 

• API testing, 

• Integration, Load, 

• Unit testing, 

• Deployment testing, 

• Checklist,

• Regression

General




Language Version

Eg: Python 3.x/ Java 7


Framework Version

Eg: Django Version


Library Version

Eg: PyMongo Version


Licenses

Apache, MIT, GPL

Others




Metrics

Deployment Frequency

% of failed Deployments

Time from Checkin to Deployment


Structured Interview Questions

There is no particular order to these structured questions. Use them to suit your hiring needs. 

As discussed in Hiring, keep in mind

  • The reason you're asking this question - what are you trying to get out of this question?
  • What does a good answer and great answer look like?
  • Good follow up questions

General
  • “If you look at what you’ve done, would you put yourself in the camp of people who say they’ve been lucky in their career?”
  • What did you learn from your mom?
  • Tell me about the best class you have ever taken. What was the class? Why was it good?
  • Other than money, what rewards, benefits, or work situations are most important for you?
  • Can you provide an instance when your ethics was tested? What was your dilemma? What decision did you take?
  • Some people consider themselves to be "big picture people" and others are detail oriented. Which are you? Give an example that illustrates your preference
  • Describe a typical day in your most recent job.
  • Most of us have more than one reason for leaving a job. What are some of yours, from past or present experience?
  • Tell me about a time when you had to practice professional confidentiality. What was the situation? What did you do? What was the outcome?
  • Describe your ideal job. Discuss the amount and type of supervision you prefer, contact and kinds of relationships with co-workers, job tasks, and freedom to work or to make decisions.
  • What is the role you are looking for ? How does this align with your aspirations?
  • What else besides your education and job experience qualifies you for this job
  • What do you think was your major role in the last company?


Motivation and Recognition
  • Give me some examples of the different ways in which you have demonstrated recognition of individual/team success?
  • How do you know if someone is not motivated?
  • Can you give me some examples of how you have motivated your team?
  • What incentives have you provided for your team?
  • How do you personally motivate yourself? What gets you enthused?
  • Describe when you worked the hardest and felt the greatest sense of achievement
  • What motivates you?
  • How will you find if a guy is not motivated?
  • How will you deal with person not performing properly?
  • How do you motivate your team? Give examples of when you did that, how did you go about and what happened as a result


Energy, Enthusiasm and Curiosity
  • How do you energise yourself to do things which don’t particularly interest you?
  • What do you know about this company and what do you think about it? Based on what you know from outside, where do you think the company should be going?
  • Do you have any questions about our business?
  • What additional activities do you currently undertake that give an indication of your enthusiasm for your job?
  • What examples can you give of your enthusiasm for what you do?
  • What have you identified as being the three major areas of focus if you were appointed to this position?
  • Give an example of a situation where you have gone above and beyond the call of duty when carrying out a specific task or project?
  • Tell me about a time when an emergency caused you to reschedule your workload or to work outside normal hours
  • What are the questions they ask?
    • "What is the most technically challenging software problem your company is facing right now?"
    • "What would you expect me to accomplish in my first 90 days?"
    • "How will my performance be evaluated?"
    • "How do your best employees grow within the organisation?"
    • "What is your company's short-term / long-term business strategy? What is your competitive advantage?"
    • "What do you (the manager) love most about your job?"
    • "What are your company's biggest weaknesses? How do you intend to improve on them?"


Initiative, Creativity, and Innovation
  • Give me some examples of where you have ‘led the field’ in your current role
  • What have you done previously which could be described as particularly innovative
  • What methods do you use to source new ideas – where does your inspiration come from?
  • Name a time when your creativity or alternative thinking solved a problem in your workplace.
  • Give me an example of when you thought outside of the box. How did it help your employer?

Leadership
  • “Describe your responsibilities as a leader.”
  • What is your proudest achievement ?
  • Give an example of when you took a risk to achieve a goal. What was the outcome?
  • Under what circumstances might you seek a co-worker or peer for advice or suggestions? Provide examples of time you did that in the past.
  • How will you improve relationship with customers, peers or team in another geography
  • What are the conflicts you've had with your peers, reportees and senior management. Why did you have them?  How did you resolved them?
  • Tell me about a time you had difficulty working with someone (can be a coworker, client). What made this person difficult to work with for you? (Follow-ups: What steps did you take to resolve the problem? What was the outcome? What could you have done differently?)
  • What are some long-range objectives that you developed in your last job? 
  • Share an example of when you established and accomplished a goal that was personally challenging. What helped you succeed?
  • Share an experience when you applied new technology or information in your job. How did it help your company?
  • Describe a time when you successfully persuaded another person to change his/her way of thinking or behaviour.
    • Describe a time when you took personal accountability for a conflict and initiated contact with the individual(s) involved to explain your actions.
    • How would you describe your leadership style?
    • What is leadership according to you?
    • If you had the authority or power, what one thing would you change about your current/last position?
    • What’s your philosophy on leadership?
    • What makes for an effective leader?
    • Name a time when your advice to management led to an improvement in your company or otherwise helped your employer.
    • Provide an example of a time when you successfully organised a diverse group of people to accomplish a task.
    • Everyone has made some poor decisions or has done something that just did not turn out right. Give an example of when this has happened to you
    • “Tell me about a project that you led that failed. Why did it fail and what did you learn?
    • If you were offered this position and you accepted, what one or two major contributions do you think you would make, in the short term (in the first few weeks) and in the long term (after a year or so)?
    • Describe your responsibilities as a leader. Where do you focus the most? Why?
    • What are the two important things that you contributed to the team. What was the outcome? Explain how it all happened?
    • Were there any occasion you pushed back on upper management or reset expectations?

      Business Acumen
      • How have you succeeded in increasing revenue?
      • What other opportunities exist for growth or improvement
      • Describe how your job relates to the overall goals of your department and organisation
      • How have you grasped the specific opportunities offered by your location?
      • What do you see as being the major areas for growth?
      • What marketing methods do you feel are the most effective and appropriate?
      • What are the important levers of your business? How do you contribute to it?
      • What would you have done differently at your last company if you had been CEO?
      • Give a problem the company actually faces and the current thinking on that problem

      Management
      • How would you describe your management style?
      • What are your areas of improvement as a manager?
      • What qualities do you think are essential in a successful manager?
      • What is the most difficult people management issue that you've dealt with?
      • What did you do on your very best day at work?
      • What does office politics mean to you, and how do you see and handle politics?
      • How has your style changed over the years?
      • How does your boss’s management style compare with your own?
      • Compare the styles of different people to whom you have reported?

      Team, People, Culture
      • In your experience, what is the key to developing a good team?
      • How do you get your staff working together as a team?
      • How do you figure out if someone is a team player? (company, team, self -- in that order)
      • How has this changed in different work environments?
      • How would members of your team/colleagues describe you?
      • Tell me about the way you have changed the behaviours and attitudes of your team?
      • How has your boss got the best out of you and your colleagues?
      • How have you demonstrated your ability to influence and persuade?
      • Give me an example of the sort of contribution you make at meetings.
      • How do you enlist the support and co-operation of your colleagues?
      • How do your colleagues see you as a member of the team?
      • How would you describe your team style?
      • Describe the most difficult/stressful problem you have encountered with a colleague and how you resolve that problem?
      • Tell me about an instance where you have sought the advice of others to help you reach a difficult decision?
      • Describe a situation in which you helped a person set a goal and then supported the person to achieve that goal. What was the goal and how did you help?
      • What are the essence of good cultural elements?
      • How do you improve and measure productivity of team ?
      • What is the most difficult people you've dealt with?
      • What are the important/toughest people management issue you have faced ?
      • Do you remember the guy in the middle of your stack rank. What are the conversations you have had with him? What are the challenges that you and he faced?
      • Have you laid off people? What process do you follow?
      • How will you hire people? What qualities do you look for?
      • Tell me about a time your behaviour had a positive impact on your team. (Follow-ups: What was your primary goal and why? How did your teammates respond? Moving forward, what’s your plan?)
      • Tell me about a time when you effectively managed your team to achieve a goal. What did your approach look like? (Follow-ups: What were your targets and how did you meet them as an individual and as a team? How did you adapt your leadership approach to different individuals? What was the key takeaway from this specific situation?)
      • “Describe a few of your peers at your company and what type of relationship you have with each of them.”
        • Instances you've changed the direction of team, the reason and the result of that intervention
        • Tell about a time when you built rapport quickly with someone under difficult conditions.

        Analytical ability, Problem Solving, Innovation and Creativity
        • What are the top 3 problems you have solved in the last year?
        • Tell me about an instance  where you have had to analyse information and then act on your findings
        • Describe a difficult problem you’ve had to deal with
        • Can you give examples of how you identified a small problem and solved it before it got out of hand
        • What would you do if you had a decision to make, for which no procedure exists?
        • Can you offer any examples of situations which have required you to find a creative or imaginative solution to a problem?
        • How could you be more creative in your present role?
        • Tell me about the last time you monitored or reviewed information and detected a problem. How did you respond?
        • Provide a time when you were able to identify a complex problem, evaluate the options, and implement a solution. How did the solution benefit your employer?
        • Instance of two innovative ideas that had a great impact
        • Describe a project or situation that best demonstrates your analytical abilities.
        • Share an experience in which your attention to detail and thoroughness had an impact on your last company.
        • Name a time when you identified strengths and weaknesses of alternative solutions to problems. What was the impact?
        • Give examples of patterns you were able to recognise and help your team avoid them.


        Self Awareness, Learning
        • Describe the last thing you did for self-improvement?
        • Have you thought about where you want to be in two, three, five years?’
        • What are your failures and learning from them?
        • Consider the five main elements of your current job and rank them in order of: Importance, Interest, Time, Personal competence
        • What are your strengths and areas of improvement?
        • What do you consider as your core competency?
        • You will have read the job description, how long do you think it will take you to become competent in this new role?
        • In what ways do you encourage feedback on your performance?
        • Give me some examples of your willingness to accept feedback for self improvement
        • Describe a situation where your work or one of your ideas was criticised?
        • Have you done the best work you are capable of doing?
        • Tell me about something you are not very proud of
        • What sort of coaching would you expect from your manager? and what coaching do you do for your team?
        • What aspect of your style of performance is your manager likely to have to spend time on developing or improving?
        • What have others been able to learn from you?
        • How do you set an example to colleagues and subordinates?
        • How would your staff/colleagues describe your personality?
        • What activity, in your working life, gives you the greatest kick?
        • What drives you mad?
        • What would you like to be remembered for?
        • How has your management style changed over the years?
        • What is your team role style?
        • What is your learning style?
        • Describe any classes, experiences, or training you have received that prepare you for this job.
        • How have you changed in the last few years?
        • What do you expect to be doing in 2 years and 5 years?
        • How have you kept up on relevant resources and information about a topic of interest to you?
        • What is the next thing you want to learn how to do, or how to do better? What is your plan for accomplishing this?
        • If you were to look into the mirror (introspect) what would you change with yourself?
        • What’s the last thing you’ve read or learnt?
        • How would describe yourself with “three adjectives”?
        • If I were to contact your reportee, what will be his feedback about you? 
        • If I were to contact your manager, what will be his feedback?
        • Why should we select you?
        • What one or two words would most of all of your previous supervisors use to describe you?
        • What is one thing that you like and one thing that you dislike about your engineering culture? What would you change?
        • What part(s) of you last/present position did you like least? What did you do to try to overcome the situation? What would you think if you know in advance that this situation would be present in your new job?
        • What are the reasons for rejecting you?

        Mentoring
        • “Can you tell me about four people whose careers you have fundamentally improved?”
        • There is a lot of talk about ‘empowering’ people.  What does this mean to you?
        • Tell me about your successes in preparing people for promotion?
        • Give me some examples of the sort of responsibilities and accountabilities which you have delegated
        • How do you monitor and control the work of your team?
        • If you have to identify a person’s suitability for promotion, what factors do you consider?
        • In what ways has your current job prepared you to take on greater responsibility?


        Resilience
        • Tell me how you have reacted to setback or disappointment which you have encountered
        • Talk me through a performance management situation you have had to deal with
        • What sort of things do you worry about?
        • How do you make your opinions known when you disagree with your manager or a colleague?
        • Tell me about a crisis situation which really challenged you – what did you do?
        • Describe the most complex disciplinary situation you have had to deal with
        • What do you do when you have a great deal of work to accomplish in a short time span – how do you react?


        Engineering and Release

        • How confident will you be with the release and based on what criterion?
        • What is the SDLC model you follow?
        • What is the complete engineering process?
        • Who are your stakeholders?
        • How do you manage 2 week releases and 2 month releases?
        • Say you have 20 people and a project that runs for an year. How will you manage it?
        • What is your release cycle?
        • Have you pushed back on any release based on Quality?
        • What are the metrics you depend on?
        • Were there any occasions you have pushed back on Product management?

        Communication
        • Why is effective communication important?
        • What methods do you currently use to ensure that your team know what’s going on?
        • What are the barriers to good communication?
        • How will you handle communication with a dispersed team?
        • How frequently do you believe you should meetings?
        • In what way do you encourage people to express their views?
        • How do you make your goals and targets clear?
        • Describe how effective you are at listening?
        • Describe a situation where you felt you had not communicated well. How did you correct the situation

        Organisational and Planning Ability
        • How do you organise yourself and plan your day
        • In determining priorities, what factors do you take into consideration?
        • Talk me through the planning of a particularly complex activity you have undertaken recently
        • How do you react to interruptions and distractions –when you are up against a deadline?
        • when you are working on something complicated?
        • during the normal course of work?
        • How do you cope under pressure and avoid stress?


        Project Management
        • When starting a new project, what are the first things you do?
        • How do you maintain an overview of a particular project?
        • If one of your suppliers fails to meet a deadline, how do you deal with the situation?
        • What are the important things to monitor during any large project?
        • How do you go about establishing realistic targets and completion dates?
        • What do you do if you have two conflicting priorities?
        • How do you measure results?


        Presentation Skills

        • What was the last presentation you gave? Or what was the last memo you wrote?
        • Share an experience in which you presented to a group. What was the situation and how did it go?
        • What sort of presentations have you undertaken in the past?
        • How have you prepared for these?
        • What visual aid do you feel give the best support in a presentation to a group of 25 people?
        • What is the largest number of people you have had to present to?
        • How do you feel prior to a formal presentation?
        • What type of presentation makes you the most nervous?
        • How do you overcome your pre-presentation nerves?

        Standards and Professionalism
        • Tell me about your ‘obsessions’
        • What methods do you use to maintain the highest possible standards?
        •  Talk me through a typical day – what is important to you?
        • What to you get on your ‘high horse’ about?
        • What do you believe is critical to the success of your work?


        Costs and Budgets
        • What is the most challenging part of budgeting for you?
        • What techniques do you use to keep costs under control?
        • How do you save money?
        • Describe your experiences in setting budgets?
        • How do you monitor and control your budgets?
        • What cost optimisations have you done?
        • But is the biggest source of leak ?
        • What are some of the low hanging fruits you have fixed? How about the tough ones?
        • Tell me about the last contract or cost estimate you approved. How about the one you rejected? Why?

        Role models
        • Who, during your career, have been your mentors and how?
        • What aspects of your former bosses’ management style have you adopted?
        • What sort of things has your manager done that you disliked?
        • What sort of things have you learned from your colleagues?
        • What do you hope your new manager will provide to benefit your personal development?
        • How frequently would you want involvement from your manager?


        Customer Care
        • How would you define your customers?
        • What do you do to ensure you always exceed your customers’ expectations?
        • Who do you believe are you a customer of?
        • What do you think is outstanding customer service?
        • How would you describe the quality of service you receive from them?

        Aspirations / Personal Development
        • What are the reasons for your success in your current job?
        • What is the ‘extra special’ quality that you will bring to this role?
        • Ideally, how would you now like to see you career develop?
        • How realistic is this ambition?
        • What do you feel this job will do to enhance your career and personal development?


        Personal, Interests
        • Tell me how you spend your spare time?
        • Why do enjoy these activities?
        • What other activities would you pursue if you had more time/money?
        • What is your dream situation?
        • How do you balance work / life?
        • How do you handle stress?