Hadoop User Group (HUG) Ireland and the Return of the Big Data Jedi

The 12th’s return by HUG Ireland in association with Oracle... A September Special for all Big Data lovers…

Whilst the team at HUG Ireland took an event moratorium in August, we have not wasted a minute in setting up a cracking Autumn 2016 schedule for our members, which forms Dublin’s premiere technical Big Data community. November and October already have confirmed speakers covering areas of Data Science and Data Pipelines that should not be missed by one and all!

We have also been busy with the team at Oracle to bring you a great lineout for September 12th at the Morrison Hotel on Ormond Quay from 6pm. Two quality speakers are speaking on the evening on The Future of Analytics, Oracle Big Data SQL and The Three I"s of Big Data with a bar on premises, so we can all catch up with each other on site on what’s new; before and after the quality presentations by Mark Rittman, CTO of Rittman Mead and Ian Sharp, Director of Data Science with Oracle UK.

So, some background on our speakers:

Mark is CTO and co-founder at Rittman Mead, an Oracle (and more recently, Cloudera) Partner specialising in BI, big data and data warehousing implementations, training and support. Mark is an Oracle ACE Director, author of two Oracle Press books and leads Rittman Mead's customer projects around Hadoop, cloud and machine learning.

Ian is a Data Scientist at Oracle with over 15 experience helping customers deliver insight and innovation around business intelligence, Data Warehousing and Big Data. He has worked across industries with organisations such as Worldpay, Sky, NHS, BT, HSBC, AVIVA, the NHS, Bord Gais and the Discovery Channel.

The agenda for the evening from 6pm looks like this:

The Future of Analytics, Data Integration and Data Driven Applications on Big Data Platforms by Mark Rittman, CTO of Rittman Mead

Up until now thought of as a cheap storage offloading layer for more capable relational data warehouses and BI tools, Hadoop is fast maturing into the default platform for all analytic and data integration workloads. In this session we'll look at how technologies such as Apache Arrow, Apache Kudu and Apache Drill will bring in-memory, column-store BI functionality to Hadoop but extend that to cover flexible-schema data held in data lakes and NoSQL databases; we'll see how Apache Spark, Apache Kafka and real-time streaming functionality coming to vendor ETL tools such as Oracle Data Integrator will enable real-time loading and integration with traditional relational databases; and finally we'll see how a new generation of e-commerce, logistics and customer applications will take advantage of the machine learning and predictive analytics capabilities provided by Hadoop to add smarts and intelligence to day-to-day operational applications.

Using Oracle Big Data SQL to add Hadoop + NoSQL to your Oracle Data warehouse by Mark Rittman, CTO of Rittman Mead

Oracle Big Data SQL 3.0 extends the Oracle SQL dialect, security framework and advanced analytics capabilities of the Oracle database to big data platforms, initially just for Oracle Big Data Appliance but now made available for commodity Cloudera and Hortonworks clusters. Complementing basic Apache Hive but competing with other next-generation SQL-on-Hadoop solutions such as Impala, Presto and Drill, Oracle Big Data SQL lets you work with big data sources as if they were another set of external tables – and can bring Exadata style query offloading to this significantly lower-cost platform. In this session we’ll explore at a technical level how Big Data SQL works (including the key limitations and “gotchas”), see how features such as storage indexes and storage handlers introduced with the 2.0 release significantly improve what the product can do, how statistical and advanced analytics features in Oracle Database Enterprise Edition can also be used on Hadoop sources, understand at a high-level what bringing Oracle security Hadoop means – and see if it’s really now possible to use Big Data SQL without the previous Exadata and BDA prerequisites.  

The Three I"s of Big Data, Insight, Innovation and Identify by Ian Sharp,  Lead Data Scientist with Oracle UK

Insight and innovation aside, Big Data Social Media is increasingly about Identity; who we are, who we aspire to be, and in the case of Fraud and Theft, who we pretend to be. Data Science, Big Data Analytics and Machine Learning have emerged as the prime technologies and disciplines to distill nuances and value around these identities. This presentation discusses the challenges and opportunities of big data analytics against polyglot sources, the evolution of new languages and frameworks such as R, Python, SciKit Learn and Zeppelin and the operational settings  around wider BI consumption, Streams and Apps APIs.  Using leading edge reference cases, we will explore  the stories and focus on the challenges for integration, governance and stability faced  in their  transition from Hadoop driven Innovation  to Mainstream business success. Finally we will highlight  some up and coming trends for the industry to watch

So, RSVP today and check out our twitter hashtag at #HUGIreland for the latest from our community.  Looking forward to seeing you all on the 12th!

Sources/Credits:

Pics:

Hadoop User Group (HUG) Ireland

Credits:

Morrison Location:

https://www.google.ie/maps/place/The+Morrison,+a+DoubleTree+by+Hilton+Hotel/@53.3466752,-6.2681787,17z/data=!3m1!4b1!4m5!3m4!1s0x48670e83401492d5:0xdc556b1fc3ff0b3b!8m2!3d53.346672!4d-6.26599

Event Page:

http://www.meetup.com/hadoop-user-group-ireland/

Speaker Profiles:

Ian:

https://www.linkedin.com/in/ian-sharp-03a3835?authType=NAME_SEARCH&authToken=-usr&locale=en_US&trk=tyah&trkInfo=clickedVertical%3Amynetwork%2CclickedEntityId%3A15847534%2CauthType%3ANAME_SEARCH%2Cidx%3A1-3-3%2CtarId%3A1471963329454%2Ctas%3Aian%20sharp

https://www.oracle.com/uk/index.html

Mark:

https://www.linkedin.com/in/markrittman?authType=NAME_SEARCH&authToken=3wfo&locale=en_US&srchid=2114422741471963406937&srchindex=1&srchtotal=3&trk=vsrp_people_res_name&trkInfo=VSRPsearchId%3A2114422741471963406937%2CVSRPtargetId%3A2087871%2CVSRPcmpt%3Aprimary%2CVSRPnm%3Atrue%2CauthType%3ANAME_SEARCH

http://www.rittmanmead.com/

Big Data - Here Today, Here Tomorrow

 Big Data Adoption, it’s really happening.. Is it time to get on board?

Technology has evolved over the last 60 years and today we continue our lightning fast journey through time with technology by our side. The term “big data” has been used by many to describe data analytics which are often compiled from spreadsheets. They say ‘(Microsoft) Excel is an accountant’s best friend’ yet it comes with a lot of limitations that cannot be overlooked in today's world. The evolving nature of current data driven decision-making at senior levels in organisations places a huge burden on departments responsible for analytics and management reports.

We as a race have produced more information in the last 2 years then we have in the last ten thousand years making data analytics in today's operating environment a real challenge.  The evolution of big data has seen technological breakthroughs in streaming, batched pipelines, distributed clusters and data centre management. If we are to capitalise on opportunities presented by big data, we need real time or near real-time streaming along with batched data ability that is indexed and easy to transform so we can analyse it producing insights in support of the management decision making process.

Here are some high level points that one should think about when considering big data as a professional and/or an enthusiast:

• Open source with it’s democratisation is truly the king of big data. Harness the power of the open source community and collaborate in the evolution of big data that will service your needs
• Big Data Architecture is the looking glass into the future. Think distributed computing in two ways, what works and what works for you. Scalability comes at a price... make sure your solution comes with the appropriate flexibility in partitioning and try to keep one data layer only in your big data stack.
• Data Management should determine your security protocols ranking your data in layers of sensitivity. Ensure parity between your policies and your big data architecture. Allow this to drive things like storage solutions (in-house, cloud, hybrid), design, deployment, etc.

Data analytics and visualisation should become more “visually orientated” with a data on demand mindset. Create a higher level of abstraction based on visual filtering for your analytics suite.

Finally, when you have a working data pipeline servicing your needs through a big data stack that presents useful tools and analysis for use, make sure you use it. Package your end result in an easy to understand format with visually driven reports from one of your most valuable resources, which is the large volumes of data you collect every day! Now is the time to transform your future with big data that will not only create awareness and understanding, it will create value for your company both now and into the future.

About the author: John Mulhall is a Marketing Technologist and newly Minted Software Developer. John is also a committed blogger and from February 2016 onwards, will be publishing blogs every second week on topics around Technology, Leadership and Sustainable Capitalism.

Sources/Credits:

Pics;

https://stocksnap.io/photo/RPUX05EF6J

https://stocksnap.io/photo/QVKE96OD36

https://www.123rf.com

Credits;

Land of Illusion - Our World View

When looking at the world through abstract lenses, patterns of life emerge that raise questions about we make sense of it

We all at some point love and hate life, see it in different shades of both extremes and seem to never question what we have come to take for granted. From the air we breathe to the evenly applied angles of petals on a daisy, it is simply life, part of our world, which allows us to achieve what we have set out to achieve within its boundary paths.

With the impressive march of technology and time, my own journey has been revealing more and more patterns in life. I was once ignorant of these patterns but now ponder them with excitement and interest. To explain my developing interest in the patterns of life, I often wondered how some folks do better than others after a paradigm shift in “their world”. Who adapts, who fails, who succeeds, who changes the picture completely once again??? This is man made and sits against a world that is more and more connected to itself.

So, enter nature and one begins to wonder about the patterns of uniformity and synchronicity… why do they happen? Who makes it happen? Is it part of nature’s “DNA” for it to happen? Maybe our world is bigger and wider than we once thought? The patterns and mechanics of nature are often mimicked by man, thus we have microwaves, computers, antibiotics, central heating and so on.. A profitable relationship wouldn’t you agree? What if we push the boundaries of this relationship with the world around us, within us, above us and beyond us?? What could we discover and seek to understand, which could help our children’s children on their journey through life?

Here are some of the things I think are gateway areas to future progress, creating a progressive technological window to tomorrow’s world today:

• Quantum Computing - Eric Ladizinsky, CEO of D-Wave gave a great talk on the current progress of quantum computing at Trinity College Dublin late last year where he was able to confirm production of 1000 qubit quantum processors. An amazing breakthrough indeed. Map the progress of this technology over the next 10 years and one can expect to see parallelisation in computer processing power increase many fold within the start of a paradigm change in computer science. This positive disruption to the technologies that run our world today will open a gateway of possibilities for tomorrow’s world! Exciting times indeed!
• DNA, Big Data and Disease - As society becomes more comfortable with the concept of mass data flows, big data analysis and genetic research, the ability to produce more and more insights leading to cures from diseases ranging from haemochromatosis to cancer will escalate the pace of medical technology innovation. At the heart of this innovation, data science will lend a critical hand in the identification of patterns and probabilities that will aid the unlocking of answers to some of mankind’s deadliest diseases.
• Artificial Intelligence - What if every robot, mainframe and distributed cluster had its own AI controller. Imagine a core that you could converse with, understand emotions, right/wrong and then execute self augmenting programming to capture a better outcome for the process. This is a flash of what could be achieved with such technology. Responsible adoption of course is key and should lead all development projects. These outcomes can create the AI reality of the future which will positively impact society at that point.
• Energy - Fossil fuels will soon come to pass and the experimentation with “alternative energy” is something to be explored by disrupters. The fact is that the iron grip of the energy industry over fuels will soon “run out of gas” as the ability to generate energy from solar protons via panels will become so effective, a new age of energy will arise in the quest to have self generating cells that last a lifetime and harness the power of nature to tend to the needs of man.
• Propulsion - We need to travel to the stars. We won’t go far on a tank of gas, so with a developing understanding of the world around us, we can expect to see that new forms of energy, navigation and propulsion will be unlocked by our intrepid minds that don’t accept that status quo as fixed. The quest for knowledge and understanding is a powerful one in a universe that carries us forward with the arrow of time allowing us via entropy to never stand still.

It’s not enough to merely survive life, we must live it to understand our time here and how we fit into the world around us. If we open our hearts and minds to the possibilities that surround us, we commune with a universe that will help us find our way turning a land of illusion into a land of hope for a better day!

Sources Credits:

Pics:

https://stocksnap.io/

https://stocksnap.io/photo/9866CDBF0B

https://stocksnap.io/photo/P6HXICOTZ5

           

Sources:

http://www.irishtimes.com/business/quantum-computers-could-be-next-great-leap-for-mankind-1.2411733

http://journals.aps.org/prl/abstract/10.1103/PhysRevLett.80.2245

http://ems.tcd.ie/

Strategic Decision Making - Cognitive Bias V Data Driven Decision Making

What gives cognitive bias a competitive edge today.. and why data driven decision making prevents it from being a liability in tomorrow’s world

As we head into 2016, it’s natural to think of the future, what it holds and how we will fair out when it arrives! Shall our children enjoy an uninterrupted run of good luck and will our lives only get better with the success we have worked so hard for?? All very natural questions to think about in the new year yet we rarely think of the strategic decisions made by others that can impact events creating missed opportunities and/or unseen dark clouds on the horizon of our life’s path.

Thinking about this strategic review of our lives, how many of us have thought strategically about the future assuming patterns of future success based on a summation of our past experiences? How many have done so without any real interrogation of the past to see if it is feasible to repeat into the future? This oversight of thought is known as cognitive bias (aka intuitive bias), which by Webster’s definition is “Common tendency to acquire and process information by filtering it through one's own likes, dislikes, and experiences.”

Cognitive bias has its origins in evolution. We created a mental filtering system to manage the large volumes of data we receive every day and through a learning process of experience, education and observation, we developed a short cut list of actions to speed up our reaction times, which to date increase our chances of survival. Whether it’s steering clear of an angry Lion on the Savanna or under pressure in the office where we “skim” through a 200 page report by reading the 2 page bullet point summary; our ability to maintain performance under pressure relies on our ability to make lightning fast decisions that rely on prediction based on past experiences. In essence, the decision we made “successfully” in a 100 near-similar cases to date will gain favour with us over a perfect match solution someone else told us about just the other day.

So, sounds like all is well? I would agree if the decision was based on 2 minutes before work to get a Mocha Latte or an Americano coffee. However, what is the case when you have a major decision to make on strategy that affects many people along with large amounts of resources in an organisation?? After all, if you have the right man in the driving seat, then trust his gut, it’s worked this far… right?? 

My considered answer is this. The “gut call” is fine to a point but our world is changing even faster than ever. We cannot rely on even successful past experiences to solely guide our future strategic decision making. We need to embrace data driven decision making by creating a process pipeline for it; where we can evaluate a data rich and clearly presented issue(s) before reaching any swift conclusions based on our experiences, likes and dislikes (aka cognitive bias, the “gut call”). Companies who embrace data driven decision making will increasingly build a competitive edge over those who don’t for the following reasons:

• Dynamic Markets - they change and trend even faster with the onset of the information era and if you don’t understand your customers wants and needs nearly at an individual, real-time level based on their behaviour, then you will lose out to the competitor who can
• Impact Awareness - the age of industrialist capitalism is coming to a clear impasse where the impact of strategic decisions on staff, internal productivity, performance, regulatory compliance and social responsibility all matter as much as the impact on revenue, which in times past (and present for some companies) was the sole consideration. The need for transparency and accurate insights by mapping the impact of strategic decisions is increasingly more important than ever
• The Disrupter Effect - the information age has levelled the playing field to a great extent for new tech start ups who have proven to be vibrant, savvy and able to disrupt pre-existing non tech industries. Apple’s disruption of the music industry is a classic example of this. Microfinance based start ups like Grid Finance shows great disruption potential to the retail arms of banks and lower level investment companies providing a direct investor - beneficiary platform at a fraction of the cost a bank or investment company would take in fees and/or equity
• Faster to Market - the onset of the information era and web 2.0 technologies (cloud) has made it possible to develop and bring to market new products in a fraction of the time to times past. The effect of this in the marketplace is felt by shortening the cycle of new product generation, which in turn is making product life-cycles shorter and more prone to disruption

Staying ahead of the information game is becoming increasingly more dynamic, harder to predict especially if the decision maker has no access to data when making the decision.That said, as the onset of technology has driven the information era and the rise of disruption. It also drives the data driven decision making approach, which companies can now use to their competitive advantage. Companies wanting to embrace it are well advised to consider the following:

• Risk Management - develop risk management structures for enterprise that cover your risks in the main areas of business, finance (including FX) and operations. Use tools like risk registers, risk mapping, scenario planning and risk assessments in all major strategy decisions.
• Data Management - a data management platform should be built and structured so as you know how to get to your data in a highly available state, retrieving what you want, when you want.
• Process Management - process management structures should be easily maintained, executed, highly effective and auditable via control points in the process flow that feeds into metrics for the business. This should be automated to the extent that it is feasible based on company size and data quality. If the control points don’t enforce data quality, then they need to be revised until they do.
• Predictive Analytics - transactional data feeding analytical data sets is one of the biggest tools one can get in the modern era. That said, if a robust process management structure does not guarantee good quality data sets, then predictive analytics will produce no qualitative analysis. The need to get the quality and structure of your data sets right from input to analysis cannot be overstressed. It’s the basis of a value creating “data pipeline.” 

Technology is marching ahead at strength in all areas above, which is why a business considering how they make their senior level decisions should be considering the above points plus more to create an infrastructure of awareness that will make the enterprise more able to meet dynamic markets and act and/or react in time to build upon prior success into an ever uncertain future. The future is here, do you use a data driven approach to decision making? Do leave a comment with your thoughts...

Sources/Credits:

Pics;

www.forbes.com

www.etlsolutions.com

www.reviewpro.com

https://stocksnap.io/photo/EC50CC8658

Credits;

Cognitive Bias: http://www.businessdictionary.com/definition/cognitive-bias.html

https://www.grid.finance/