Skip to main content

Posts

Productivity improvement for remote teams!

The typical working hours in IT companies are from 10 am to 6 pm, though it could extend beyond this time depending on the nature of the project. Usually, we expect everyone to put in about 8 hours a day. There are two broad categories to classify these eight hours: Collaboration time and Core working time.  Collaboration time is when interactions with others are needed and includes all the client meetings, standups, team huddles, and discussions. Ideally, these are the hours that enable individuals to complete their work. Individuals in the team have limited choices on when these meetings have to happen as it could involve multiple stakeholders. Core working time is when the actual work gets done and is the productive hours of the individual. The more focused the individual is, the more effective they are.  These two times overlap with regular office working hours and are not conducive to peak productivity. Some teams strive to have dedicated Core working hours when there are no
Recent posts

Why Cloudformation template over boto3?

Why Cloudformation template is preferred over boto3 to provision infrastructure in AWS? 1. Zero cost. There is no cost for using the CloudFormation template to provision resources. On the other hand, if the resources are provisioned using boto3 in lambda, execution does cost money. Even if the price is negligible, every penny helps. 2. Scaleability. If the resource provision takes more than 15 minutes, it is impossible to provide the resource using lambda directly. In provisioning large resources with heavy bootstrap installations, it could be possible to hit those limits. 3. Fail-Safe. When the resource provision fails for any reason, lambda doesn't take responsibility to clean up the resources instantiated so far unless it is handled explicitly. On the other hand, CloudFormation Stack does a clean rollback by reverting to the initial state. It does save money if the resources are cleaned up appropriately. 4. Less error-prone. There is only one way (declarative using JSON or YAML)

Import 1 billion records from Oracle to HDFS in a record time

The problem: A large scale manufacturing organization aggregates data from different sources, maintains it in a single Oracle table, and the number of records is in the order of a little over a billion. A monthly process has to fetch the data from Oracle to HDFS.  The constraint: Ideally, only the difference for each month could be fetched. But, there is little to no control over the Oracle data source and there is no reliable way to identify the delta. Hence, all the data have to be fetched all the time. To give a perspective, if the table is exported as a CSV from a SQL Client (say, SQL Developer), it takes more than 20 hours to download the table. The tool: Sqoop is the standard tool used to import data from the relational database to HDFS. The solution: $ sqoop import -D **oracle.row.fetch.size=50000 --fetch-size 15000 --num-mappers 40** --table ` <schema>.<table_name> ` -connect ` <jdbc_connection_url> `   --username ` <user> ` -P --target-dir ` <hdfs_ta

Six ways to land rovers on Mars.

Six ways to land robotic rovers on Mars Mars Rover problem is a popular problem statement used by companies to check object orientation and test-driven development skills. In this article, we'll take the core problem statement and see how the solution evolves through six different levels. Knowledge of high school level maths and little python helps to follow this article. The actual Problem Statement: A squad of robotic rovers is to be landed by NASA on a plateau on Mars. This plateau, which is curiously rectangular, must be navigated by the rovers so that their on-board cameras can get a complete view of the surrounding terrain to send back to Earth. A rover's position is represented by a combination of x and y coordinates and a letter representing one of the four cardinal compass points. The plateau is divided up into a grid to simplify navigation. An example position might be 0, 0, N, which means the rover is in the bottom left corner facing North. In order t

Counting

Letters and numbers are the two eyes of man. - Thirukkural (392) Numbers, the oldest and greatest invention by the man kind, are the base for measurements, without which no useful inventions could ever be made. The objective of this post is to think through the representation of measurements, particularly the counting, in ancient tamil culture. Ancient tamil culture : Measurements are broadly classified into 7 parts. Counting ( எண்ணல்) Weight ( நிறுத்தல்) Liter for liquid ( முகத்தல்) Volume for solid ( பெய்தல் ) Length ( நீட்டல்) Time ( தெறிப்பு) Comparison ( சார்த்தல்) Let's limit the scope only to counting for this post. Counting are generally classified into two parts Chitrilakkam - Fractions. Perilakkam - Whole Numbers. The following shows the names given for the fractions: The following shows the names given for the whole numbers: Other measurements include 100 crores is Kumbam ( கும்பம் ) 1 lakh crores is Sangam ( சங்கம் ) 1

The human synergy

After nearly two and half years of inactiveness, I'm reloaded back to throw more ramblings in the open space of internet. The plan is to have a weekly journal on an idea or a product, that I understand or learnt that week. Let's get started and this week's cynosure is "reCAPTCHA" We, often, see sites that ask us to enter the content of a distorted or skewed image of letters or numbers, that is known as CAPTCHA. The widely known fact is that it enables the site to distinguish a human from any automated bots or scripts. It is so reliable, that vast number of sites are using it, and about 200 million captchas are answered by humans in a day. Roughly it takes about 10 seconds per person to answer a captcha. In summation, each day more than 150,000 hours of human effort is consumed by these Captchas, which does nothing more than confirming that the detail is entered by a human. Could this human effort be used for a higher purpose? Yes. The answer

Coupling

A good design thrives for low coupling . The lowest form of coupling is set of disconnected objects, which couldn't form a system. (Since, objects don't interact with each other.) The highest form of coupling is Inheritance, which is one of the pillars of OOPS. The answer for this irony can be "everything is a trade-off in software development". Principles are only guidelines and not rules. An Anology: "Drive Slowly" is a common driving safety guideline. At the slowest speed (0 km/hr), the vehicle doesn't move. Also, people generally wants to buy a vehicle which gives maximum power and speed. Its a guideline to be applied pragmatically. P.S: My another post which shows the legal violation of Command Query Separation Principle .