Spacy - NLTK: Language detection - nltk

Question

I am currently working on a project dealing with a bunch of social media posts.
Some of these posts are in English and some in Spanish.
My current code runs quite smoothly. However, I am asking myself does Spacy/NLTK automatically detect which language stemmer/stopwords/etc. it has to use for each post (depending on whether it is an English or Spanish post)? At the moment, I am just parsing each post to a stemmer without explicitly specifying the language.
This is a snippet of my current script:
import re
import pandas as pd
!pip install pyphen
import pyphen
!pip install spacy
import spacy
!pip install nltk
import nltk
from nltk import SnowballStemmer
from nltk.corpus import stopwords
from nltk.stem.porter import PorterStemmer
!pip install spacy-langdetect
from spacy_langdetect import LanguageDetector
!pip install textblob
from textblob import TextBlob
# Download Stopwords
nltk.download('stopwords')
stop_words_eng = set(stopwords.words('english'))
stop_words_es = set(stopwords.words('spanish'))
# Import Stemmer
p_stemmer = PorterStemmer()
#Snowball (Porter2): Nearly universally regarded as an improvement over porter, and for good reason.
snowball_stemmer = SnowballStemmer("english")
dic = pyphen.Pyphen(lang='en')
# Load Data
data = pd.read_csv("mergerfile.csv", error_bad_lines=False)
pd.set_option('display.max_columns', None)
posts = data.loc[data["ad_creative"] != "NONE"]
# Functions
def get_number_of_sentences(text):
sentences = [sent.string.strip() for sent in text.sents]
return len(sentences)
def get_average_sentence_length(text):
number_of_sentences = get_number_of_sentences(text)
tokens = [token.text for token in text]
return len(tokens) / number_of_sentences
def get_token_length(text):
tokens = [token.text for token in text]
return len(tokens)
def text_analyzer(data_frame):
content = []
label = []
avg_sentence_length = []
number_sentences = []
number_words = []
for string in data_frame:
string.join("")
if len(string) <= 4:
print(string)
print("filtered")
content.append(string)
avg_sentence_length.append("filtered")
number_sentences.append("filtered")
number_words.append("filtered")
else:
# print list
print(string)
content.append(string)
##Average Sentence Lenght
result = get_average_sentence_length(nlp(string))
avg_sentence_length.append(result)
print("avg sentence length:", result)
##Number of Sentences
result = get_number_of_sentences(nlp(string))
number_sentences.append(result)
print("#sentences:", result)
##Number of words
result = get_token_length(nlp(string))
number_words.append(result)
print("#Words", result)
content, avg_sentence_length, number_sentences, number_words = text_analyzer(
data["posts"])

Answer 1

Short answer is no, neither NLTK nor SpaCy will automatically determine the language and apply appropriate algorithms to a text.
SpaCy has separate language models with their own methods, part-of-speech and dependency tagsets. It also has a set of stopwords for each available language.
NLTK is more modular; for stemming there is RSLPStemmer (Portuguese), ISRIStemmer (Arabic), and SnowballStemmer (Danish, Dutch, English, Finnish, French, German, Hungarian, Italian, Norwegian, Portuguese, Romanian, Russian, Spanish, Swedish).
When you determine the language of a post through spacy_langdetect, the next thing you should do is explicitly instruct to use the appropriate SpaCy language model or NLTK module.

Answer 2

Use GoogleTrans Library for this
#/usr/bin/python
from googletrans import Translator
translator = Translator()
translator.detect('이 문장은 한글로 쓰여졌습니다.')
This Returns
<Detected lang=ko confidence=0.27041003>
So, this is the best way to do so if you have an internet connection and is better in most cases than Spacy as Google Translate is more mature and has better algorithms, ;)