How StockTwits Uses Machine Learning to Make Better Products

Thomas Ott -

Fascinating behind the scenes interview of StockTwit's Senior Data
Scientist Garrett Hoffman.

He shares great tidbits on how StockTwits uses machine learning for
sentiment analysis. I've summarized the highlights below:

  • Idea generation is a huge barrier for active trading
  • Next gen of traders uses social media to make decisions
  • Garrett solves data problems and builds features for the StockTwits
    platform
  • This includes: production data science, product analytics, and
    insights research
  • Understanding social dynamics makes for a better user experience
  • Focus is to understand social dynamics of StockTwits (ST) community
  • Focuses on what's happening inside the ST community
  • ST's market sentiment model helps users with decision making
  • Users 'tag' content for bullish or bearish classes
  • Only 20 to 30% of content is tagged
  • Using ST's market sentiment model increases coverage to 100%
  • For Data Science work, Python Stack is used
  • Use: Numpy, SciPy, Pandas, Scikit-Learn
  • Jupyter Notebooks for research and prototyping
  • Flask for API deployment
  • For Deep Learning, uses Tensorflow with AWS EC2 instances
  • Can spin up GPU's as needed
  • Deep Learning methods used are Recurrent Neural Nets, Word2Vec, and
    Autoencoders
  • Stays abreast of new machine learning techniques from blogs,
    conferences and Twitter
  • Follows Twitter accounts from Google, Spotify, Apple, and small tech
    companies
  • One area ST wants to improve on is DevOps around Data Science
  • Bridge the gap between research/prototype phase and embedding it into
    tech stack for deployment
  • Misconception that complex solutions are best
  • Complexity ONLY ok if it leads to deeper insight
  • Simple solutions are best
  • Future long-term ideas: use AI around natural language