Hyper-Personalized Experiences Leveraging Artificial Intelligence Techniques in Collaborative Filtering and Deep Neural Networks.
Recommender or recommendation systems are everywhere -from Spotify suggesting music based on listeners’ preferences to Amazon nudging buyers to add more to the shopping cart; that is “People who bought this also bought…” feature. Recommendation systems have led us to expect personalization everywhere in our lives. Employees today expect the same personalized experience from their employers as Netflix, Spotify or Amazon. They need to seamlessly transition to a workday the path set by their weekend experiences; namely searching for a movie to watch on Netflix. Nowhere in the business world is personalized recommendation experience more critical than for learning and skill-building.
Recommender systems have several moving parts. The most important aspect being the component that picks a small list of items from a large catalog of options. The key to a good recommendation is relevance-a topic of deep study and debate over the past few decades.
Secrets of Recommendations
One commonly used technique is collaborative filtering which is based on user interactions. Collaborative filtering uses explicit ratings from the users for a piece of content and implicit ratings based on the user interactions (e.g., viewing a content although not explicitly rating it) as well as other user attributes (e.g., other users they follow). The core idea is to extrapolate recommendations based on behaviors exhibited by others in similar situations.
Developing robust collaborative filtering has been a long and arduous journey. In 2006 Netflix started a competition to improve its recommendations that was further refined in 2009 for the best collaborative filtering algorithm. The $1 million winner in 2010 was never used and the competition discontinued- a testament to the difficulty in making improvements in relevance.
Fast forward to 2021, a popular technique in use today is hybrid filtering that combines content-based filtering with collaborative filtering to produce better recommendations. The most relevant of the predictions are based on whether a user will like (or interact with) a suggested item. We have seen great progress in optimizing collaborative filtering algorithms using advanced techniques such as matrix factorization which we will explore quickly.
In a matrix factorization model, a simple inner product of user and content feature embeddings are used. As such, the performance of the algorithm is dependent on the assumption of data linearity. However, in reality, the relationship between users and content is much more complex. The linear dot product does not adequately capture the complex non-linear relationships that represent user’s preferences. In other words, complex relations between users and their interactions can cause the recommendations to be significantly off-requiring a much more nuanced approach to solving the problem.
Neural Networks
This is where neural networks come in – with several layers of decisions adding up to a final recommendation weighing several nuances at a time. This added complexity not only produces better recommendations but also gets to the outcomes faster.
The application of neural networks has been steadily on the rise in recent years with the abundance and declining costs– from self-driving cars to virtual assistants, neural networks have been the driving force behind much of the explosive growth of applications and hype around machine learning. Although deep neural networks have been in use in the likes of image and speech recognition to great results, recommendation systems have not had the benefits of its application until recently.
Collaborative Filtering
It has been good to see researchers looking at finding different ways of applying neural network based learning to improve collaborative filtering in recent years. Since neural networks are very good at finding latent variables, especially in large and heterogeneous datasets, they are a natural fit for collaborative filtering improvement. One architecture approach that is getting good traction is NCF (Neural Collaborative Filtering). It should be noted that it is important to experiment with both deep and wide neural networks while applying to collaborative filtering.
Neural Network-based learning technique, however, does come at a cost. If not being careful, we could introduce over-fitting of the data, especially in case of sparse datasets of user ratings. As such, it is very important to use hyperparameter tuning with regularization techniques such as weight decay and dropout.
In the domain of Learning Experience and Skill-Building, businesses look for relevance in a rapidly changing environment. A recommendation engine plays a key role in providing the learning experience for the users zeroing in on signals from others in the organization, and the characteristics of the content or skill-building activities to be performed.. As the “Spotify of learning” for an organization, EdCast aggregates content from both internal and external sources and provides personalized, relevant, contextual, and timely content to a learner for upskilling and reskilling. Like any good supervised/semi-supervised learning algorithm, the predictions get better as more relevant data is generated, consumed in the flow of work. The innovations at EdCast are designed to continue to innovate on the underlying learning, skilling and career 
growth algorithms.
At EdCast we are committed to provide a revolutionary talent experience, empowering learning, skilling and career-mobility in the flow of work. Innovations in AI are a critical component of traversing the large and exponentially growing volume of content available/generated in the company, industry groups, open source communities, experts, educational institutions and enthusiasts on the internet. Aggregating and curating this to build skills is a tough challenge for any organization.
Talk to us to learn how we can help you leverage AI and machine learning to revolutionize your employee experience.
#AiAsAService #AiInTheEnterprise #datascience #neuralNetwork #SpotifyOfLearning
References:
- He, X.; Liao, L.; Zhang, H.; Nie, L.; Hu, X.; Chua, T.-S. Neural Collaborative Filtering. In Proceedings of the 26th International Conference on World Wide Web Companion, Perth, Australia, 3–7 April 2017;
- Deep Learning Architecture for Collaborative Filtering Recommender Systems – Jesus Bobadilla , Santiago Alonso and Antonio Hernando – 3 April 2020
