Home

I'm looking for:

We're using cookies, but you can turn them off in your browser settings. Otherwise, you are agreeing to our use of cookies. Learn more in our Privacy Policy

Data science and AI: A guide for Investment Managers

colorful montage image with people standing on a circuit board
Published 24 May 2024

Data science and artificial intelligence (AI) are vital tools for improving productivity and accelerating growth and innovation. How can investment managers best cultivate and leverage them, and what are the common pitfalls to avoid?

Investment managers use data to understand what happened in the past and to try to predict what will happen in the future. They can do so with increasing precision as data becomes ever more plentiful and accessible, and the techniques to organize and analyze it more robust and powerful. This is driving a revolution in productivity, decision-making and innovation in finance and virtually every other industry.

The volume of data created has been growing exponentially in recent decades, first with the rapid expansion of the internet, and then with the proliferation of Internet of Things (IoT) devices, sensors, and other machines that generate or gather data. This has created a rich and ever-expanding pipeline of data that can be used to generate valuable investment insights.

What are data science and AI?

Observing the explosion of data, British mathematician Clive Humby declared in 2006 that “data is the new oil.

Like oil, data’s potential is only realized when it is refined and processed. By pairing statistical techniques with specific subject-matter expertise in a process known as data science, meaning and insights are extracted by uncovering patterns in data. These patterns can then be used to drive decisions and make predictions.

A typical data science lifecycle involves the following six steps:

  • Identifying a problem to be solved and understanding the business objective
  • Collecting relevant raw data
  • Cleaning and processing the data to ensure it is accurate and usable
  • Exploring the data to identify patterns
  • Constructing and evaluating models to make predictions and forecasts based on the data
  • Communicating the findings to stakeholders and deploying the models to solve problems

The line between data science and AI is blurred. Both data science and AI, at their core, are about finding patterns in data. AI goes further than data science by using patterns found in data to accomplish tasks that are traditionally associated with human rather than machine intelligence, such as visual perception, speech recognition, automated decision-making, and the creation of various forms of content.

Why are data science and AI important for investment professionals?

The explosion and democratization of data brought about by the internet and digitalization have meant that investors can no longer rely on privileged access to information to drive investment outperformance.

As investment firms have incorporated data science and AI into almost everything they do (see Figure 1), there is a growing need to instill at least a basic familiarity with these topics across their entire workforces, rather than leaving it to technology or data science teams to handle all the data. In short, it is now imperative for investment professionals to be proficient in data science.

Figure 1: Data Science and AI are Applied Across Most Functions at Investment Firms What business activities has your organization been routinely applying to AI and Big Data to? (Select all that apply) 0% 50% 100% Compliance Streamlining other mid-/back-office operations Customer service Decision making in our core business Cybersecurity Sales and marketing Risk management Data analysis in our core business CFA Institute, Handbook of Artificial Intelligence and Big Data Applications in Investments Source:

Failure to achieve this means growing volumes of data will go unexploited. Bringing portfolio managers into the process can yield considerable value because they have a better sense of which data is relevant to their business lines and can provide the foundation for effective exploration and model building.

The investment industry appears to be headed to a future in which the entire organization is involved in filtering, managing and analyzing data. A 2022 CFA Institute survey revealed that the most in-demand type of talent across firms is finance professionals with AI and big data skills (see Figure 2).

To learn more about these topics, please read:

Figure 2: Relevance of Data Skills in Hiring Decisions A. Over the next two years, which type of talents is your organization primarily looking to hire more? (select up to two) 0% 5% 10% 15% 20% 25% 30% 35% 40% AI and big data talents with no finance skills AI and big data talents with some finance skills Talents with both finance and AI and big data skills Finance talents with some AI and big data skills Finance talents with no AI and big data skills B. What business activities has your organization been routinely applying AI and big data to? (select all that apply) 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Compliance Streamlining other mid-/back-office operations Customer service Decision making in our core business Cybersecurity Sales and marketing Risk management Data analysis in our core business CFA Institute, Handbook of Artificial Intelligence and Big Data Applications in Investments. Source:

How AI can supercharge data analysis

AI is an indispensable tool to closing the gap in the growing volumes of data within organizations that go unused for analytics.

AI will help streamline and automate coding and other data science-related tasks. Rather than making data science jobs obsolete, AI will transform them. It will free data scientists to broaden and improve their knowledge and skills, including learning more about the businesses they support.

In addition, AI will make it possible to extract the rich potential of the “unstructured” information that accounts for up to 90% of data: from sensors and credit cards, web scrapes, newsfeeds, and social media posts. According to a recent CFA Institute survey, investment analysts are already using three main types of unstructured data:

  • Individual data, such as social media, blogs, product reviews, web search trends, and cellphone location data
  • Business data, such as data on credit card transactions, store visits and bills of lading
  • Satellite data, which gives insights into agricultural trends, rig activity, road traffic, ship locations and mining data

Most investment managers expect to incorporate such alternative data sources more deeply in their decision-making process (see Figure 3). Although alternative data sets are unlikely to replace traditional sources entirely, they are seen as complementary, giving angles that might otherwise be missed.

Figure 3: Alternative Data are Expected to Become a More Important Driver of Investment Decisions Respondents’ Answers to the Question, “How Important Do You Expect Each Of the Sources Of Information to Be In the Future For Your Investment Decision Making?” Sustainability ratings Alternative data sources (social media, web scraping, credit card data, etc.) Proprietary, in-house metrics Your professional network Market sentiment indicators Independent research providers Large financial markets data vendors (e.g., Bloomberg, FactSet) Companies' financial statements Credit ratings 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 76% 15% 5% 4% 72% 20% 3% 5 % 50% 40% 5% 5% 43% 50% 5% 2% 33% 56% 7% 3% 31% 57% 10% 2% 23% 66% 10% 2% 13% 77% 8% 3% 10% 67% 19% 3% More important Stay the same Less important Not applicable CFA Institute Skills and Careers Survey. Source:

Advances in natural language processing (NLP), a subset of AI that enables machines to interpret and analyze human language, have been a game changer in exploiting alternative data sources. In particular, NLP is used in asset management.

  • Track sentiment analysis
  • Extract themes from blocks of text and speech
  • Uncover risks in corporate filings
  • Prioritize sales efforts and gather business intelligence

To explore these topics in greater depth, please see:

Why communication skills are integral to using data effectively

Investment professionals and data scientists will need to work more closely together and get better at communicating with each other.

Such cooperation and communication will help avoid common pitfalls encountered when attempting to extract patterns and insights from data. Although algorithms and machines may make better data-driven decisions than humans because they are not subject to fatigue, emotions and biases, they, too, can make mistakes and find relationships between variables where none exist as a result of missteps in data selection and model design. Errors and biases inadvertently introduced by humans into the data science and AI pipeline could then be magnified by machines.

These mistakes can be remedied, but they must first be identified. That is where human judgement comes in, as well as cooperation and communication among colleagues to avoid being led astray by data. Since launching a faulty model can prove extremely costly in the investment world, team members need to work together to thoroughly question and challenge each other’s models before deploying them in the real world.

For more information on these topics, please read:

What are the ethical concerns when handling data?

Investment managers also need to be wary of ethical concerns surrounding data science and AI by striving to responsibly handle data and develop and deploy algorithms.

Without such safeguards, big data and AI have the potential to perpetuate societal bias, violate privacy, go against the interest of clients and lead to losses.

As financial institutions press ahead with large investments in developing their data science and AI capabilities, it is imperative that they formulate and adopt comprehensive and relevant governance frameworks.

The four pillars of an ethical decision framework for AI are:

  • Data integrity
  • Accuracy
  • Transparency and interpretability
  • Accountability

By adhering to these principles, investment firms cannot only reduce their chances of falling foul of upcoming regulation, but also help establish sufficient trust in their technology among consumers and lawmakers to enable it to spread quickly.

After all, when it comes to data science and AI, it is becoming increasingly clear that financial institutions – as with firms in other industries– stand the best chance of doing well by doing right.

For an in-depth discussion of this topic, please see:

View more Data & Technology stories

You may also be interested in

Want to learn more about Data science?

With the advent of data science, the investment industry is changing rapidly. Investment firms are facing new challenges and these changes will have implications for your career.

Learn more
Abstract data science illustration