- Introduces some history about the Big Data problem
- How it affected some of the silos technologies like RDBS
- How they solve the scalability issue
Thursday, March 6, 2014
Disruptive Possibilities: How Big Data Changes Everything
Thursday, December 26, 2013
My best decision of 2013
- Culture: I wanted to work for a company who looked at IT as an innovation engine rather than a cost center.
- Team: I wanted to work with awesome engineers (very motivated, smart, and with a lot of experience). Also, I wanted to be the dumbestperson in the room.
- Tackle big harry audacious goals: To be part of a company whose products revolve around big problems that would (without a doubt) have an impact to the bottom line.
I was introduced to 3Ci in 2007 when I was trying to sell a startup that I was working for by the name of Up-Mobile. The startup wasn’t doing well in the US market, so I was handed the task of selling our customer based to another company. During that time, I met with 3Ci’s senior management and some of their architects. Their company was doing some similar things as our startup, unlike us though, they were able to evolve quick and found a potential market with top tier companies and a compelling product. Many of 3Ci programmers were very involved in open source projects. For good or bad, I was attached to Up-Mobile and I wasn’t ready to make a move. Nevertheless, 3Ci left a great impression. Later down the line, I became the organizer of the Miami Java User Group (MJUG) and Miami JVM Meetup. 3Ci was more than helpful to provide a venue, food, and beverage with no strings attached. There was a big turnaround of 3Ci employees when I did presentations. It turned out that they also do a lot of presentations on things that they are doing. Also, they are committed to open source projects while keeping away from the golden hammer anti pattern - if you have a hammer, everything looks like a nail. They had the motto of using the best tool for the problem.
Culture
- Build a sustainable company in the emerging market for enterprise mobile services
- Create a great culture that focuses on the personal and professional development of our team
- Do important work for quality clients
I came from a couple of small companies or startups (with less than 20 people). When you work for a small company, you not only know everyone’s name, but you know what everyone is doing and why. Although 3Ci has more than 100 people, and many of them are distributed, everyone on the team understands what the goals are because of these three rules. The culture is what promotes the exceptional people working there, the proactive and ownership of tasks, breed result-focused, and the team. 3Ci's culture have created a set of values based on this culture:
We are not in the business for the business alone but for a higher purpose - to make lives better, to solve important problems, and to enjoy what we do. Our Data Scientist, Oliver, once said, “Don’t live a life to do a great work. Live a great life, and then the great work will follow”.
3Ci strives to be one of the best places you will ever work.
Time and time again, you see that we become either the best place in South Florida to work or America’s most promising company.
We’re in it together
We’re champions of change
We need to change quickly and adapt. Anyone with some experience knows this, but the one that are constantly thinking are those entrepreneurs that work at startups. I learned early on my career that the life a technology company is composed of three: you either go big, stay small, or be eaten. This was mentioned by Mike FitzGibbon (Fitz), president and cofounder, at the company's "All Hands Call", we need to be customer-centric and be able to quickly and effectively execute goals.
Team
One of the main engineers at 3Ci during 2007 was Alejandro (Alex), and I met him on an open source project (Kannel). I checked some of his code, and I really like what he was doing. But the best things about him was his curiosity. He was a system administrator at 3Ci, yet he was learning Java at that time and committing to Kannel. I was also a good friend with the one of the main committers and founder of Kannel, Stipe Tolj. Stipe was also working for 3Ci as a consultant and knew Alex personally. He told me about some of the things that the company was doing, and I was intrigued to say the least. After a couple of e-mails, Alex and I pretty much hit it off. At that time he was working in Europe, then he was promoted and moved to 3Ci (Boca Raton, FL). The moment that I found out, I immediately sent him my resume. During my interview, I met a couple of guys: Mauricio and Carlos (Carlitos). Two of their best architects. This is another way of knowing how good the company, they had a good technical interview process. The interview was hard but I did well, so I was hired. I’ve never looked back. I highly enjoy it, and it’s the reason of why I make the 1.5 hour commute on Tuesday and Thursday from Kendall to Boca Raton.One of my favorite quotes is "Hire the most amazing people you can. Communicate goals. Turn them loose. Profit." - Sam Schillance, Box. One of my rules of working at a job is to be surrounded by great engineers, then see how much they complain about my code. This is by far why I love this company. Again, It’s not the graffiti, nor the free food (which is delicious), nor the fact that I work remotely from Miami. It’s the engineers that I work with. When I started, I met a few great engineers. People from my team are former Googlers, people who worked at Bloomberg, current Apache committers, and entrepreneurs. Just a great set of interesting people who love to code and yet find a way to find a life after work - teachers, band players, great parents, rock climbers, etc. For the past nine months, I’ve learn more than any other company in my professional career, both personally and technically. Don’t get me wrong, it’s a very competitive company! The bar is raised high the moment you are hired, and the expectations are tremendous, but the safety net of the team is why I have a smile on my face every time I come to work. For example, there are five people that I always turn to in case I have any questions: Gabe, Alex, Tyler, Oliver, and Rob. In my view, these guys composed the core of the technology of 3Ci. They have a tons of experience, are highly innovated, smart, charismatic, and a great set of guys. Together, I call them my G.A.T.O.R team and it’s such a pleasure working with them.
My final interview was with one of the founders and COO, Mark Smith. One of his questions was, “What do you want to do in the future?”. I told him that I wanted to eventually start my own company, learn Machine Learning, and keep running MJUG and Miami JVM Meetup. He said, “Awesome, we’ll help you”. Done! After that, I was asking “Where do I sign?" Mark and Alex are by far the best leaders that I ever had. They know when to help the team, and when to get out of their way. If you worked for more than 10 years, then you’ll know that it’s hard to find a good company, even harder to find a good leader that could be a mentor.
Big Hairy Audacious Goals
Another huge incentive for me was to be able to tackle what James Collins calls, “Big Hairy Audacious Goals”. Although I can’t say anything about the goals for the company, trust me, they are as big as they are interesting. This is what motivates me. The fact that I work with an awesome team tackling some really big problems, help set the path of pushing the envelop as much as we can. Because of this, we look at other technologies and think outside the norm. I believe that there is a challenge-to-great-developers matrix
. In my experience, great engineers gravitate to hard challenges. This is where I want to be. I don’t want to work on another CRUD application (been there, done that, got the t-shirt!) I believe that the best combination of retention of great engineers is to have great challenges for them to solve, have a great culture, and provide the best atmosphere for them.
I’ve always want to work for a company like 3Ci and I’m glad to be part of it. It’s hard to find companies such as these in South Florida. 3Ci is committed to hire the best. They are also very much focused on the culture. The best way that I can describe its culture is using this quote, “When you combine a culture of discipline with an ethic of entrepreneurship, you get the magical alchemy of great performance" - Jim Collins, Good to Great. I once read that you should choose a profession that you enjoy and that serves as many people as possible. Focusing on serving others - not on building wealth. Serve well and money will follow. I’m glad that this year I was able to scratch that one from my “to-do” list.
Tuesday, June 4, 2013
Functional Thinking Video Review
Neal points out the major advantages of using functional programming:
- Language Evolution: all major languages are adding functional features.
- Results over steps: create optimized applications to solve a problem rather than using frameworks
- Immutability: the freedom of not worrying about the state of the objects - "failure atomicity"
- First-class/higher-order functions
- Pure Functions
- Recursion
- Strict Evaluation
Friday, May 31, 2013
Configure MTR in Mac
First Install brew
Install MTR using the following commands:
Unfortunately, the configuration installs in the /usr/local/sbin
What I end up doing is removing the symbolic link and point it to my bin
Thursday, March 7, 2013
Personal MBA Book Review
Educating yourself about anything is a Tao - there's no end to the process. The journey itself is the reward.If you are the type of person that wants to learn more about business, I highly recommend it. The book distilles a series amount of great books into one. Here's what you learn:
- how businesses work
- how people work (chapter 6 -8)
- how systems work (chapter 9-11)
- Manager and leadership overload.
- Finance and accounting.
- Financial intelligence for entrepreneurs
- How to read a financial report
- Quantitative analysis: thinking statistics, turning numbers into knowledge
I found it kind of sad that the book didn't covered any quantitative analysis in this analytic age, but I was happy that he covered how systems work. This is the chapter that really resonated with me. Here are my notes on the systems chapters.
Analyzing Systems
Deconstruction: the only way to analyze a complex system is by deconstructing it. "Deconstruction is the process of separating complex systems into the smallest possible sybsystems in order to understand how things work." You need to find the "ins" and "outs" of the system for you to understand it. Unable to understand complex systems, you need to decompose them into smaller systems for you to understand them.Measurement: You need to measure the systems so you know if it's doing fine or not. "Measurement helps us avoid the absence blindness when analyzing systems. Remember: we have a hard time seeing things that aren't present. Measuring different parts of a system in operation helps to identify potential issues before they arise." One example is diabetes, a person needs to measure its blood glucose to know if it's too high or too low. A person cannot tell the level of glucose unless he/she measures it.
Key Performance Indicator (KPI): some measurements are better than others. Key performance indicator tell you exactly the status of your system. "Key performance indicators are measurements of the ciritical parts of a system. Measurements that don't help you make improvements to your system are worse than worthless: they are a waste of your limited attention and energy." Few questions the author used to identify business' KPI:
- Value Creation: how quickly is the system creating value? What is the current level of inflows?
- Marketing: how many people are paying attention to your offer? How many prospects are giving you permission to provide more information?
- Sales: how many prospects are becoming paying customers? What is the average customer's lifetime value?
- Value Delivery: how quickly can you server each customer? What is your current returns or complaints rate?
- Finance: what is your profit margin? How much purchasing power do you have? Are you financially sufficient?
Analytical Honesty: always look into data without discrimination of personal feeling. "Analytical honesty means measuring and analyzing the data you have dispassionately. Since humans are social creatures, we tend to care deeply about how other perceive us, which give us a natural incentive to make things look better than they actually are. If you purpose is to actually make things better, this tendency can get in the way of collecting accurate data and conducting useful analysis.
Sampling: is the process of taking at random a small percentage of the total output, then using it as a proxy for the entire system. If you ever had blood taken at the doctor's office, you'll have a good idead of what sampling entails.
Margin of Error: this entails on the percentage of accuracy your system has based on tests. "Is an estimate of how much you can trust your conclusions from a given set of observed samples." When it comes to analytical confidence, more data is always better.
Ratio: is a method of comparing two measurements against each other. By diving your results by your input, you can measure all sorts of useful relationships between different parts of your system. Here are some useful ratios to track:
- Return on promotions: for every $1 you spend in advertising, how much revenue do you collect?
- Profit from Employee: for every person you employ, how much profit does your business generate?
- Closing Ratio: for every prospect you serve, how many purchase?
- Returns/Complains Ratio: for every sale you make, how many choose to return or complain?
- mean: average
- median: sort the values in order of high to low, then finding the quantity of the data point in the middle of the range. Median are actually a specific form of analysis called a percentile: the median is the value that expresses the fiftieth percentile. By definition, 50% of the values in the set will be below the median.
- mode: is the value that occurs most frequently in a set of data. Modes are useful for finding clusters of data - a set can have multiple modes, which can alert you to potentially interesting interdependencies in the system that produced that data.
- midrange: is the value halfway between the highest and lowest data points i a set of values. To calculate the midrange, add the highest and lowest values, then divide by two. Midranges are best used for quick estimates - they're fast, and you only need to know two data points, but they can be easily skewed by outliers that are abnormally high or low - Bill Gate's bank balance.
Correlation and Causation: causation is a complete chain of cause and effect. Correlation is not causation. Just because you can here illegal doesn't make you a criminal. Causation is always more difficult to prove than correlation. When analyzing complex systems with many variables and interdependencies, it's often extremely difficult to find true causality.
Norms: are measures that use historical data as a tool to provide context for current measurements. If you are selling Christmas ornaments, it's a waste to compare them Q4 with Q2. You should compare this Q4 with the previous Q4.
Proxy: measures one quantity by measuring something else. Used with care, proxies can help you measure the measurable - just be sure your proxy is directly and highly correlated with the subject of interest.
Segmentation: is a technique that involves splitting a dataset into well-defined subgroups to add additional context. Finding out that last quarter's sales increased by 20% is good, but knowing that they were done by 80% of women is even better. There are three common ways to segment customer data: past performance, demographics, and psychographics.
- Past performance: segments customers by past known action. For example, you can segment customer sales data using previous sales data, comparing sales to new customers with sales to customers who have previously purchase from you. Lifetime value calculations are a form of segmentation by past performance.
- demographics segments customers by external personal characteristics. Personal information like age, gender, income, nationality, and location can help you determine which customers are your probable purchasers.
- psychographics: segments customers by internal psychological characteristics. Typically discovered via surveys, assessments, or focus groups.
Improving Systems:
Optimization: this is the process of maximizing the output of a system or minimizing a specific input the system requires to operate. Optimization typically revolves around the systems and processed behind your "key performance indicators", which measure the critical elements of the system as a whole. Improve your KPIs, and your system will perform better.Refactoring: the process of re-engineering the systems to improve efficiency without changing its output. The primary benefit of refactoring isn't improving the output - it's making the system itself faster or more efficient.
The Critical Few: this is the 80/20 rule from Vilfredo Pareto.
For best results, focus on the critical inputs that produce most of the results you want.Finding the inputs that produce the outputs you want, then make them the focus of your time and energy. Ruthlessly weed out the rest.
Diminishing Return: after a certain point, having more of something can actually be detrimental. In the book the author explains about his work at P&G. At this company, he was the one in charge of analyzing the results of his advertising (tv spots). After time, the commercial will "wear out". That's the effect of diminishing return. "Optimize and refacgtor up to the point you start experiencing diminishing returns, then focus on doing something else".
Friction: Every business process has some amount of friction. The key is to identify areas where friction currently exists, then experiment with small improvements that will reduce the amount of friction in the system.
Automation: refers to a system or process that can operate without human intervention.
Find a way to automate your system, and you open the doors to scale via duplication and multiplication, improving your ability to create and deliver value to more paying customers.
The Paradox of Automation: the more efficient the automated system, the more crucial the contribution of the human operators of that system. When an error happens, operators need to identify and fix the situation quickly or shut the system down - otherwise, the automated system will continue multiplying the error.
The Irony of Automation: the more reliable the system, the less human operators have to do, so the less they pay attention to the system while it's in operation. Remember the Mackworth Clock and vigilance studies conducted on British radar operators in WWII from our discussion on Novelty? Humans get bore extremely quickly if things stay the same, and the more reliable the system, the more things stay the same.
Keep your automated system's operators mentally engaged, and they'll be far more likely to notice when errors inevitably occur.
Standard Operating Procedure (SOP): is a predefined process used to complete a task or resolve a common issue. Business systems often include repetitive tasks or resolve a common issue. Well-defined standard operating procedures are useful because they reduce friction and minimize will power depletion. Instead of wasting valuable time and energy solving a problem that has already been solved many times before, a predefined SOP ensures that you spend less time thrashing and more time adding value.
Don't let your standard operating procedure lapse into bureaucracy. Remember, the purpose of an SOP is to minimize the amount of time and effort it takes to complete a task or solve a problem effectively. If the SOP requires effort without providing value, it's friction.
Checklist: a checklist is an externalized, predifined standard operating procedure for completing a specific task. Checklist will help you define a system for a process that hasn't yet been formalized. Second, using checklists as a normal part of working can help ensure that you don't forget to handle important steps that are easily overlooked when things get busy.
Checklist can produce major improvements in your ability to do quality work, as well as your ability to delegate work effectively. By taking the time to explicitly describe and track your progress, you reduce the likelihood of major errors and oversights, as well as prevent willpower depletion associated with figuring out how to complete the same task over and over again.
Cessation: is the choice to intentionally stop doing something that's counterproductive. In The One-Straw Revolution, Masanobu Fukuoka wrote about his experiments with natural farming, which mostly involved letting nature take its course and intervening as little as possible. Instead of trying to do too much, Fukuoka only did what was absolutely necessary. As a result, his fields were consistently among the most productive in the are.
Cessation takes guts. It's often unpopular to unpalatable to do nothing even if doing nothing is actually the right solution.
Resilience: what business needs is more turtles and fewer tigers. Turtles are nature's tank - they can eat different food, go through hibernation if times are tough, and shelter from an enemy if needed. Tigers depends on their strength, speed, and pray to live. If prey becomes scarce or they lose their hunting powers due to age or injury, death takes them quickly and mercilessly. This is why turtles live longer than tigers.
Resilience is a massively underrated quality in business. Having the toughness and flexibility to handle anything life throws at you is a major asset that can save your skin.
What makes a business resilient:
- low (preferably zero) outstanding debt
- low overhead, fixed costs, and operating expenses
- substantial cash reserves for unexpected contingencies
- multiple independent product/industries/lines of business
- flexible workers/employees who can handle many responsibilities well
- no single points of failure
- fail-safes/backup systems for all core process
Planning for resilience as well as performance is the hallmark of good management.Fail-safe: is a backup system design to prevent or allow recovery from a primary system failure. As much as possible never have a single critical point of failure.
Stress testing: is the process of identifying the boundaries of a system by simulating specific environmental conditions.
Scenario Planning: is the process of systematically constructing a series of hypothetical situations, then mentally simulating what you would do if they occurred. Most large business use scenario planning as the basis of a practice called "hedging": purchasing various forms of insurance to reduce the risk of unfavorable future events.
Don't waste time trying to predict an unknowable future - construct the most likely scenarios and plan what you'll do if they occur, and you'll be prepared for whatever actually happens.
The Middle Path: is the ever-changing balance point between too little and too much - just enough.
The Experimental Mind-set: constant experimentation is the only way you can identify what will actually produce the result you desire. Often, the best (or only) way to learn things is to jump in and try.
You learn the most from what doesn't go well. As long as your mistakes don't kill you, paying attention to what doesn't work can give you useful information you can use to discover what does.
Thursday, February 28, 2013
Using Deadbolt for Authorization in Your Play 2 Site With Scala
I needed security in my site. I needed a feature that would guarantee for only authorized users to go into some of the controllers (links) of my site. For non-registered users, I wanted to display a "non authorized" page so they could either login or register. I also needed for Play 2.1 using Scala 2.10. My search lead me to deadbolt 2 (D2). In this post I will cover the following:
- How to include deadbolt into your Play site
- Create a security service to validate registered users
- Send non-registered users to a "not authorized" page
- Add security to your controller
- How to validate registered users from the template
Include deadbolt into your Play site
The first thing that you need is to include the deadbolt plugin into your site. You need to include it inside your project/Build.scala. Here you will find that I added the play-plugin-mailer already, postgresql, and finally deadbolt. I had to include the resolvers for the application to work.
Add Security Service to Your Application
Before you implement the service, you need to create a subject. Subject is what determines a user in deadbolt. It also handles privileges such as roles and permissions. Below is the code for a User inside the app/models package:
Now, that we have a subject, we need to implement the DeadboltHandler to validate registered users/subjects and to send the non-register users to a "non-register" page. The main application that takes care of user validation is the getSubject.
Here the code tries to get the the username from the HTTP session, and in case there is a user, return a subject (User) with that session name. Otherwise, return a "None". When returning None, Deadbolt automatically knows that this is not a valid/registered user.
Send Non-Registerd users to a "Non-Registered" Page
For those users that are not registered (None Subject), the "onAuthFailure" will take effect.
Below is the code inside the app/security package.
Add Security to the Controller
Now, that the DeadboltHandler is implemented, you have to make some decisions based on the business rules of your site. For example, if you have an Contact Directory page where only registered users could see it, you need to assure that there is a Subject (registered user) present.
Noticed that I am also passing a new MyDeadboltHandler to the page. This is because I also need to implement some business rules inside the template. In this site, I needed to show a "Login" link for those non-registered users or display a "Contact Directory" and a "Logout" link for those authenticated users. Below is my navigation template.
Inside the "subjectNotPresent" we will handle the case when the user is not logged in. Whereas, in the subjectPresent we will show the Logout and the Contact Directory link pages.
For those controllers that doesn't require the a subject to be present, I still had to pass the implemented deadbolt handler because it had to be in every one of my pages. This is to validate if the user is authenticated.
Friday, February 22, 2013
Non-Blocking Async with Scala 2.10 and Play 2.1
In my application, I needed to consume an "expensive" (+3 seconds to respond) web service. The results had to be as fast as possible without halting the system/process. What I needed, was to process the web service in the background (thread), then display the results in the application. I used Twitter and GitHub as an example.
Then display the results with the following render.scala.html page:
At the beginning of the application, I got the following error:
The solution is to import the following:



