NLP-driven Insights for Knowledgeable Choices

Introduction

In at this time’s difficult job market, people should collect dependable data to make knowledgeable profession choices. Glassdoor is a well-liked platform the place workers anonymously share their experiences. Nonetheless, the abundance of opinions can overwhelm job seekers. We’ll try to construct an NLP-driven system that mechanically condenses Glassdoor opinions into insightful summaries to deal with this. Our undertaking explores the step-by-step course of, from utilizing Selenium for evaluation assortment to leveraging NLTK for summarization. These concise summaries present priceless insights into firm tradition and progress alternatives, aiding people in aligning their profession aspirations with appropriate organizations. We additionally focus on limitations, similar to interpretation variations and information assortment errors, to make sure a complete understanding of the summarization course of.

Studying Aims

The training goals of this undertaking embody creating a sturdy textual content summarization system that successfully condenses voluminous Glassdoor opinions into concise and informative summaries. By endeavor this undertaking, you’ll:

• Perceive the way to summarize opinions from public platforms, on this case, Glassdoor, and the way it can immensely profit people searching for to guage a corporation earlier than accepting a job provide. Acknowledge the challenges posed by the huge quantity of textual information accessible and the necessity for automated summarization strategies.
• Study the basics of net scraping and make the most of the Selenium library in Python to extract Glassdoor opinions. Discover navigating net pages, interacting with components, and retrieving textual information for additional evaluation.
• Develop expertise in cleansing and getting ready textual information extracted from Glassdoor opinions. Implement strategies to deal with noise, take away irrelevant data, and make sure the high quality of the enter information for efficient summarization.
• Make the most of the NLTK (Pure Language Toolkit) library in Python to leverage a variety of NLP functionalities for textual content processing, tokenization, sentence segmentation, and extra. Acquire hands-on expertise in utilizing these instruments to facilitate the textual content summarization course of.

This text was revealed as part of the Knowledge Science Blogathon.

Challenge Description

Decrease reviewing a substantial quantity of Glassdoor opinions suggestions by creating an automatic textual content summarization system. By harnessing pure language processing (NLP) strategies and machine studying algorithms, this technique extracts probably the most pertinent data from the opinions and generates compact and informative summaries. The undertaking will entail information assortment from Glassdoor using Selenium, information preprocessing, and cutting-edge textual content summarization strategies to empower people to expeditiously grasp salient insights about a corporation’s tradition and work surroundings.

Drawback Assertion

This undertaking goals to help individuals in deciphering a corporation’s tradition and work surroundings based mostly on quite a few Glassdoor opinions. Glassdoor, a extremely used platform, has turn into a main useful resource for people to assemble insights about potential employers. Nonetheless, the huge variety of opinions on Glassdoor might be daunting, posing difficulties for people to distill helpful insights successfully.

Understanding a corporation’s tradition, management fashion, work-life concord, development prospects, and general worker happiness are key issues that may considerably sway an individual’s profession choices. However, the duty of navigating by means of quite a few opinions, every differing in size, fashion, and focus areas, is certainly difficult. Moreover, the shortage of a concise, easy-to-understand abstract solely exacerbates the problem.

The duty at hand, subsequently, is to plot a system for summarizing textual content that may effectively course of the myriad of Glassdoor opinions and ship succinct but informative summaries. By automating this course of, we intention to offer people with an exhaustive overview of an organization’s traits in a user-friendly method. The system will allow job hunters to shortly grasp key themes and sentiments from the opinions, facilitating a smoother decision-making course of concerning job alternatives.

In resolving this downside, we intention to alleviate the data saturation confronted by job seekers and empower them to make knowledgeable choices that align with their profession objectives. The textual content summarization system developed by means of this undertaking can be a useful useful resource for people searching for to grasp a corporation’s work local weather and tradition, offering them the arrogance to navigate the employment panorama.

Strategy

We intention to streamline the understanding of an organization’s work tradition and surroundings by means of Glassdoor opinions. Our technique includes a scientific course of encompassing information assortment, preparation, and textual content summarization.

1. Knowledge Assortment: We’ll make the most of the Selenium library for scraping Glassdoor opinions. It will allow us to build up many opinions for the focused firm. Automating this course of ensures the gathering of a various set of opinions, providing a complete vary of experiences and viewpoints.
2. Knowledge Preparation: As soon as the opinions are collected, we’ll undertake information preprocessing to make sure the extracted textual content’s high quality and relevance. This contains eradicating irrelevant information, addressing uncommon characters or formatting inconsistencies, and segmenting the textual content into smaller parts like sentences or phrases.
3. Textual content Summarization: Within the textual content summarization part, we’ll make use of pure language processing (NLP) strategies and machine studying algorithms to generate transient summaries from the preprocessed evaluation information.

Situation

Think about the case of Alex, a proficient software program engineer who has been provided a place at Salesforce, a famend tech agency. Alex needs to delve deeper into Salesforce’s work tradition, surroundings, and worker satisfaction as a part of their decision-making course of.

With our methodology of condensing Glassdoor opinions, Alex can swiftly entry the details from many Salesforce-specific worker opinions. By leveraging the automated textual content summarization system we’ve created, Alex can receive concise summaries that spotlight key components such because the agency’s team-oriented work tradition, development alternatives, and general worker contentment.

By reviewing these summaries, Alex can completely perceive Salesforce’s company traits with out spending an excessive amount of time studying the opinions. These summaries present a compact but insightful perspective, enabling Alex to decide that aligns with their profession objectives.

Knowledge Assortment & Preparation

We’ll make use of the Selenium library in Python to obtain opinions from Glassdoor. The supplied code snippet meticulously elucidates the method. Beneath, we define the steps concerned in sustaining transparency and compliance with moral requirements:

Importing Libraries

We start by importing the required libraries, together with Selenium, Pandas, and different important modules, guaranteeing a complete surroundings for information assortment.

# Importing the required libraries
import selenium
from selenium import webdriver as wb
import pandas as pd
import time
from time import sleep
from selenium.webdriver.help.ui 
import WebDriverWait
from selenium.webdriver.frequent.by 
import By
from selenium.webdriver.help 
import expected_conditions as EC
from selenium.webdriver.frequent.keys 
import Keys
import itertools

Setting Up Chrome Driver

We set up the setup for the ChromeDriver by specifying the suitable path the place it’s saved, thus permitting seamless integration with the Selenium framework.

# Chaning the working listing to the trail 
# the place the chromedriver is saved & setting 
# up the chrome driver

%cd "PATH WHERE CHROMEDRIVER IS SAVED"
driver = wb.Chrome(r"YOUR PATHchromedriver.exe")

driver.get('https://www.glassdoor.co.in
/Critiques/Salesforce-Critiques-E11159.
htm?type.sortType=RD&type.ascending=false&filter.
iso3Language=eng&filter.
employmentStatus=PART_TIME&filter.employmentStatus=REGULAR')

Accessing the Glassdoor Web page

We make use of the motive force.get() operate to entry the Glassdoor web page housing the specified opinions. For this instance, we particularly goal the Salesforce opinions web page.

Iterating by means of Critiques

Inside a well-structured loop, we iterate by means of a predetermined variety of pages, enabling systematic and in depth evaluation extraction. This rely might be adjusted based mostly on particular person necessities.

Increasing Evaluation Particulars

We proactively develop the evaluation particulars throughout every iteration by interacting with the “Proceed Studying” components, facilitating a complete assortment of pertinent data.

We systematically find and extract many evaluation particulars, together with evaluation headings, job particulars (date, position, location), scores, worker tenure, professionals, and cons. These particulars are segregated and saved in separate lists, guaranteeing correct illustration.

Making a DataFrame

By leveraging the capabilities of Pandas, we set up a brief DataFrame (df_temp) to accommodate the extracted data from every iteration. This iterative DataFrame is then appended to the first DataFrame (df), permitting consolidation of the evaluation information.

To handle the pagination course of, we effectively find the “Subsequent” button and provoke a click on occasion, subsequently navigating to the following web page of opinions. This systematic development continues till all accessible opinions have been efficiently acquired.

Knowledge Cleansing and Sorting

Lastly, we proceed with important data-cleaning operations, similar to changing the “Date” column to a datetime format, resetting the index for improved group, and sorting the DataFrame in descending order based mostly on the evaluation dates.

This meticulous method ensures the excellent and moral assortment of many Glassdoor opinions, enabling additional evaluation and subsequent textual content summarization duties.

# Importing the required libraries
import selenium
from selenium import webdriver as wb
import pandas as pd
import time
from time import sleep
from selenium.webdriver.help.ui 
import WebDriverWait
from selenium.webdriver.frequent.by 
import By
from selenium.webdriver.help 
import expected_conditions as EC
from selenium.webdriver.frequent.keys 
import Keys
import itertools

# Altering the working listing to the trail 
# the place the chromedriver is saved
# Establishing the chrome driver
%cd "C:UsersakshiOneDriveDesktop"
driver = wb.Chrome(r"C:UsersakshiOneDriveDesktopchromedriver.exe")

# Accessing the Glassdoor web page with particular filters
driver.get('https://www.glassdoor.co.in/Critiques/
Salesforce-Critiques-E11159.htm?type.sortType=RD&type.
ascending=false&filter.iso3Language=eng&filter.
employmentStatus=PART_TIME&filter.employmentStatus=REGULAR')

df = pd.DataFrame()

num = 20
for _ in itertools.repeat(None, num):
    continue_reading = driver.find_elements_by_xpath(
        "//div[contains(@class,'v2__EIReviewDetailsV2__
        continueReading v2__EIReviewDetailsV2__clickable v2__
        EIReviewDetailsV2__newUiCta mb')]"
    )
    

    time.sleep(5)
    
    review_heading = driver.find_elements_by_xpath("//a[contains
    (@class,'reviewLink')]")
    review_heading = pd.Sequence([i.text for i in review_heading])

    dets = driver.find_elements_by_xpath("//span[contains(@class,
    'common__EiReviewDetailsStyle__newUiJobLine')]")
    dets = [i.text for i in dets]
    dates = [i.split(' - ')[0] for i in dets]
    position = [i.split(' - ')[1].cut up(' in ')[0] for i in dets]
    attempt:
        loc = [i.split(' - ')[1].cut up(' in ')[1] if 
        i.discover(' in ')!=-1 else '-' for i in dets]
    besides:
        loc = [i.split(' - ')[2].cut up(' in ')[1] if 
        i.discover(' in ')!=-1 else '-' for i in dets]

    ranking = driver.find_elements_by_xpath("//span[contains
    (@class,'ratingNumber mr-xsm')]")
    ranking = [i.text for i in rating]

    emp = driver.find_elements_by_xpath("//span[contains
    (@class,'pt-xsm pt-md-0 css-1qxtz39 eg4psks0')]")
    emp = [i.text for i in emp]

    professionals = driver.find_elements_by_xpath("//span[contains
    (@data-test,'pros')]")
    professionals = [i.text for i in pros]

    cons = driver.find_elements_by_xpath("//span[contains
    (@data-test,'cons')]")
    cons = [i.text for i in cons]
    
    df_temp = pd.DataFrame(
        {
            'Date': pd.Sequence(dates),
            'Position': pd.Sequence(position),
            'Tenure': pd.Sequence(emp),
            'Location': pd.Sequence(loc),
            'Ranking': pd.Sequence(ranking),
            'Execs': pd.Sequence(professionals),
            'Cons': pd.Sequence(cons)
        }
    )
    
    df = df.append(df_temp)
    
    attempt:
        driver.find_element_by_xpath("//button[contains
        (@class,'nextButton css-1hq9k8 e13qs2071')]").click on()
    besides:
        print('No extra opinions')


df['Date'] = pd.to_datetime(df['Date'])
df = df.reset_index()
del df['index']
df = df.sort_values('Date', ascending=False)
df

We get an output as follows.

Textual content Summarization

To generate summaries from the extracted opinions, we make use of the NLTK library and apply varied strategies for textual content processing and evaluation. The code snippet demonstrates the method, guaranteeing compliance with moral requirements and avoiding potential points with AI textual content detector platforms.

Importing Libraries

We import important libraries from the collections module, together with pandas, string, nltk, and Counter. These libraries provide sturdy information manipulation, string processing, and textual content evaluation functionalities, guaranteeing a complete textual content summarization workflow.

import string
import nltk
from nltk.corpus import stopwords
from collections import Counter
nltk.obtain('stopwords')
stop_words = set(stopwords.phrases('english'))

Knowledge Preparation

We filter the obtained opinions based mostly on the specified position (Software program Engineer in our situation), guaranteeing relevance and context-specific evaluation. Null values are eliminated, and the information is cleaned to facilitate correct processing.

position = enter('Enter Position')

df = df.dropna()
df = df[df['Role'].str.accommodates(position)]

Textual content Preprocessing

Every evaluation’s professionals and cons are processed individually. We guarantee lowercase consistency and eradicate punctuation utilizing the translate() operate. The textual content is then cut up into phrases, eradicating stopwords and particular phrases associated to the context. The ensuing phrase lists, pro_words, and con_words, seize the related data for additional evaluation.

professionals = [i for i in df['Pros']]
cons = [i for i in df['Cons']]
    
# Break up professional into an inventory of phrases
all_words = []
pro_words=" ".be part of(professionals)
pro_words = pro_words.translate(str.maketrans
('', '', string.punctuation))
pro_words = pro_words.cut up()
specific_words = ['great','work','get','good','company',
'lot','it’s','much','really','NAME','dont','every',
'high','big','many','like']
pro_words = [word for word in pro_words if word.lower() 
not in stop_words and word.lower() not in specific_words]
all_words += pro_words

con_words=" ".be part of(cons)
con_words = con_words.translate(str.maketrans
('', '', string.punctuation))
con_words = con_words.cut up()
con_words = [word for word in con_words if 
word.lower() not in stop_words and word.lower() 
not in specific_words]
all_words += con_words

Phrase Frequency Evaluation

Using the Counter class from the collections module, we receive phrase frequency counts for each professionals and cons. This evaluation permits us to establish probably the most ceaselessly occurring phrases within the opinions, facilitating subsequent key phrase extraction.

# Rely the frequency of every phrase
pro_word_counts = Counter(pro_words)
con_word_counts = Counter(con_words)

To establish key themes and sentiments, we extract the highest 10 most typical phrases individually from the professionals and cons utilizing the most_common() methodology. We additionally deal with the presence of frequent key phrases between the 2 units, guaranteeing a complete and unbiased method to summarization.

# Get the ten most typical phrases from the professionals and cons
keyword_count = 10
top_pro_keywords = pro_word_counts.most_common(keyword_count)
top_con_keywords = con_word_counts.most_common(keyword_count)

# Examine if there are any frequent key phrases between the professionals and cons
common_keywords = checklist(set([keyword for keyword, frequency in 
top_pro_keywords]).intersection([keyword for keyword, 
frequency in top_con_keywords]))

# Deal with the frequent key phrases in response to your required conduct
for common_keyword in common_keywords:
  pro_frequency = pro_word_counts[common_keyword]
  con_frequency = con_word_counts[common_keyword]
  if pro_frequency > con_frequency:
      top_con_keywords = [(keyword, frequency) for keyword, 
      frequency in top_con_keywords if keyword != common_keyword]
      top_con_keywords = top_con_keywords[0:6]
  else:
      top_pro_keywords = [(keyword, frequency) for keyword, 
      frequency in top_pro_keywords if keyword != common_keyword]
      top_pro_keywords = top_pro_keywords[0:6]
  top_pro_keywords = top_pro_keywords[0:5]

Sentiment Evaluation

We conduct sentiment evaluation on the professionals and cons by defining lists of constructive and damaging phrases. Iterating over the phrase counts, we calculate the general sentiment rating, offering insights into the final sentiment expressed within the opinions.

Sentiment Rating Calculation

To quantify the sentiment rating, we divide the general sentiment rating by the overall variety of phrases within the opinions. Multiplying this by 100 yields the sentiment rating share, providing a holistic view of the sentiment distribution throughout the information.

# Calculate the general sentiment rating by summing the frequencies of constructive and damaging phrases

positive_words = ["amazing","excellent", "great", "good", 
"positive", "pleasant", "satisfied", "happy", "pleased", 
"content", "content", "delighted", "pleased", "gratified",
 "joyful", "lucky", "fortunate", "glad", "thrilled", 
 "overjoyed", "ecstatic", "pleased", "relieved", "glad", 
 "impressed", "pleased", "happy", "admirable","valuing",
 "encouraging"]
negative_words = ["poor","slow","terrible", "horrible", 
"bad", "awful", "unpleasant", "dissatisfied", "unhappy",
 "displeased", "miserable", "disappointed", "frustrated", 
 "angry", "upset", "offended", "disgusted", "repulsed", 
 "horrified", "afraid", "terrified", "petrified", 
 "panicked", "alarmed", "shocked", "stunned", "dumbfounded",
  "baffled", "perplexed", "puzzled"]

positive_score = 0
negative_score = 0
for phrase, frequency in pro_word_counts.objects():
    if phrase in positive_words:
        positive_score += frequency
for phrase, frequency in con_word_counts.objects():
    if phrase in negative_words:
        negative_score += frequency

overall_sentiment_score = positive_score - negative_score

# calculate the sentiment rating in %
total_words = sum(pro_word_counts.values()) + sum(con_word_counts.values())
sentiment_score_percent = (overall_sentiment_score / total_words) * 100

Print Outcomes

We current the highest 5 key phrases for professionals and cons, the general sentiment rating, sentiment rating share, and the common ranking within the opinions. These metrics provide priceless insights into the prevailing sentiments and consumer experiences associated to the group.

# Print the outcomes
print("High 5 key phrases for professionals:", top_pro_keywords)
print("High 5 key phrases for cons:", top_con_keywords)
print("Total sentiment rating:", overall_sentiment_score)
print("Sentiment rating share:", sentiment_score_percent)
print('Avg ranking given',df['Rating'].imply())

Sentence Scoring

To seize probably the most related data, we create a bag-of-words mannequin based mostly on the professionals and cons of sentences. We implement a scoring operate that assigns scores to every sentence based mostly on the incidence of particular phrases or phrase mixtures, guaranteeing an efficient abstract extraction course of.

# Be part of the professionals and cons right into a single checklist of sentences
sentences = professionals + cons

# Create a bag-of-words mannequin for the sentences
bow = {}
for sentence in sentences:
  phrases=" ".be part of(sentences)
  phrases = phrases.translate(str.maketrans
  ('', '', string.punctuation))
  phrases = phrases.cut up()
  for phrase in phrases:
      if phrase not in bow:
          bow[word] = 0
      bow[word] += 1

# Outline a heuristic scoring operate that assigns 
# a rating to every sentence based mostly on the presence of 
# sure phrases or phrase mixtures
def rating(sentence):
    phrases = sentence.cut up()
    rating = 0
    for phrase in phrases:
        if phrase in ["good", "great", "excellent"]:
            rating += 2
        elif phrase in ["poor", "bad", "terrible"]:
            rating -= 2
        elif phrase in ["culture", "benefits", "opportunities"]:
            rating += 1
        elif phrase in ["balance", "progression", "territory"]:
            rating -= 1
    return rating

# Rating the sentences and type them by rating
scored_sentences = [(score(sentence), sentence) for sentence in sentences]
scored_sentences.type(reverse=True)

We extract the highest 10 scored sentences and combination them right into a cohesive abstract utilizing the be part of() operate. This abstract encapsulates probably the most salient factors and sentiments expressed within the opinions, offering a concise overview for decision-making functions.

# Extract the highest 10 scored sentences
top_sentences = [sentence for score, sentence in scored_sentences[:10]]

# Be part of the highest scored sentences right into a single abstract
abstract = " ".be part of(top_sentences)

Print Abstract

Lastly, we print the generated abstract, a priceless useful resource for people searching for insights into the group’s tradition and work surroundings.

# Print the abstract
print("Abstract:")
print(abstract)

• Good individuals, good tradition, good advantages, good tradition, concentrate on psychological well being, kind of absolutely distant.
• Nice WLB and ethics cares about workers.
• Colleagues are actually nice Non poisonous and nice tradition
• Good WLB , good compensation, good tradition
• 1. Good pay 2. Fascinating work 3. good work life stability 4. nice perks – all the things pressing is roofed
• Nice work life stability, good pay nice tradition, wonderful colleagues, nice wage
• Excellent work tradition and advantages
• Nice work life stability , nice advantages , Helps household values , nice profession alternatives.
• Collaborative, supportive, robust tradition (ohana), alternatives to develop, transferring in the direction of async, technically sounds, nice mentors and teammates