wordEncoding

단어를 인덱스로 매핑하는 단어 인코딩 모델

설명

단어 인코딩은 단어집에 있는 단어를 숫자형 인덱스로 매핑합니다.

문서를 단어 개수 또는 n-gram 개수로 구성된 행렬로 인코딩하려면 encode를 사용하십시오.

생성

구문

enc = wordEncoding(documents)

enc = wordEncoding(words)

enc = wordEncoding(documents,Name,Value)

설명

enc = wordEncoding(documents)는 documents에 있는 단어에서 단어 인코딩을 만듭니다.

enc = wordEncoding(words)는 단어로 구성된 배열에서 단어 인코딩을 만듭니다.

enc = wordEncoding(documents,Name,Value)는 하나 이상의 이름-값 쌍의 인수를 사용하여 옵션을 추가로 지정합니다. 예를 들어 'Order','frequency'는 빈도 수가 더 높은 단어에 더 낮은 인덱스를 할당합니다.

입력 인수

documents — 입력 문서

tokenizedDocument 배열

입력 문서로, tokenizedDocument 배열로 지정됩니다.

words — 입력 단어

string형 벡터 | 문자형 벡터 | 문자형 벡터로 구성된 셀형 배열

입력 단어로, string형 벡터, 문자형 벡터 또는 문자형 벡터로 구성된 셀형 배열로 지정됩니다. words를 문자형 벡터로 지정할 경우 이 함수는 인수를 단일 단어로 처리합니다.

데이터형: string | char | cell

이름-값 인수

선택적 인수 쌍을 Name1=Value1,...,NameN=ValueN으로 지정합니다. 여기서 Name은 인수 이름이고 Value는 대응값입니다. 이름-값 인수는 다른 인수 뒤에 와야 하지만, 인수 쌍의 순서는 상관없습니다.

R2021a 이전 릴리스에서는 각 이름과 값을 쉼표로 구분하고 Name을 따옴표로 묶으십시오.

예: 'Order','frequency'는 문서 내의 전체 빈도 수의 내림차순으로 인덱스를 정렬합니다.

Order — 인덱스 정렬

'first-seen' (디폴트 값) | 'frequency'

인덱스 정렬로, 'Order'와 함께 다음 중 하나가 쉼표로 구분되어 지정됩니다.

• 'first-seen' – 단어가 문서에 나타나는 순서대로 단어에 인덱스를 할당합니다.

• 'frequency' – 문서 내의 전체 빈도 수에 따라 내림차순으로 정렬된 단어에 인덱스를 할당합니다.

'Order'가 'frequency'이고 여러 단어의 빈도 수가 같으면 이 함수는 특정 순서로 인덱스를 할당하지 않습니다.

MaxNumWords — 인코딩할 단어의 최대 수

Inf (디폴트 값) | 양의 정수

인코딩할 단어의 최대 수로, 양의 정수 또는 Inf로 지정됩니다. 이 함수는 먼저 'Order' 옵션에 따라 인덱스를 정렬한 다음, 상위 MaxNumWords개 단어를 인코딩합니다. MaxNumWords가 Inf인 경우 이 함수는 입력 문서의 모든 단어를 인코딩합니다.

속성

NumWords — 모델 내 고유한 단어 개수

음이 아닌 정수

모델 내 고유한 단어 개수로, 음이 아닌 정수로 지정됩니다.

Vocabulary — 모델 내 고유한 단어

string형 벡터

모델 내 고유한 단어로, string형 벡터로 지정됩니다.

데이터형: string

객체 함수

ind2word	인코딩 인덱스를 단어에 매핑하기
word2ind	단어를 인코딩 인덱스에 매핑하기
isVocabularyWord	단어가 단어 임베딩 또는 인코딩에 포함되었는지 테스트

예제

단어 인코딩 만들기

예제 데이터를 불러옵니다. 파일 sonnetsPreprocessed.txt에는 셰익스피어 소네트의 전처리된 버전이 들어 있습니다. 파일에는 한 줄에 하나씩 소네트가 들어 있으며 단어가 공백으로 구분되어 있습니다. sonnetsPreprocessed.txt에서 텍스트를 추출하고, 추출한 텍스트를 새 줄 문자에서 문서로 분할한 후 그 문서를 토큰화합니다.

filename = "sonnetsPreprocessed.txt";
str = extractFileText(filename);
textData = split(str,newline);
documents = tokenizedDocument(textData);
documents(1:10)

ans = 
  10x1 tokenizedDocument:

    70 tokens: fairest creatures desire increase thereby beautys rose might never die riper time decease tender heir might bear memory thou contracted thine own bright eyes feedst thy lights flame selfsubstantial fuel making famine abundance lies thy self thy foe thy sweet self cruel thou art worlds fresh ornament herald gaudy spring thine own bud buriest thy content tender churl makst waste niggarding pity world else glutton eat worlds due grave thee
    71 tokens: forty winters shall besiege thy brow dig deep trenches thy beautys field thy youths proud livery gazed tatterd weed small worth held asked thy beauty lies treasure thy lusty days say thine own deep sunken eyes alleating shame thriftless praise praise deservd thy beautys thou couldst answer fair child mine shall sum count make old excuse proving beauty succession thine new made thou art old thy blood warm thou feelst cold
    65 tokens: look thy glass tell face thou viewest time face form another whose fresh repair thou renewest thou dost beguile world unbless mother fair whose uneard womb disdains tillage thy husbandry fond tomb selflove stop posterity thou art thy mothers glass thee calls back lovely april prime thou windows thine age shalt despite wrinkles thy golden time thou live rememberd die single thine image dies thee
    71 tokens: unthrifty loveliness why dost thou spend upon thy self thy beautys legacy natures bequest gives nothing doth lend frank lends free beauteous niggard why dost thou abuse bounteous largess thee give profitless usurer why dost thou great sum sums yet canst live traffic thy self alone thou thy self thy sweet self dost deceive nature calls thee gone acceptable audit canst thou leave thy unused beauty tombed thee lives th executor
    61 tokens: hours gentle work frame lovely gaze every eye doth dwell play tyrants same unfair fairly doth excel neverresting time leads summer hideous winter confounds sap checked frost lusty leaves quite gone beauty oersnowed bareness every summers distillation left liquid prisoner pent walls glass beautys effect beauty bereft nor nor remembrance flowers distilld though winter meet leese show substance still lives sweet
    68 tokens: let winters ragged hand deface thee thy summer ere thou distilld make sweet vial treasure thou place beautys treasure ere selfkilld forbidden usury happies pay willing loan thats thy self breed another thee ten times happier ten ten times thy self happier thou art ten thine ten times refigurd thee death thou shouldst depart leaving thee living posterity selfwilld thou art fair deaths conquest make worms thine heir
    64 tokens: lo orient gracious light lifts up burning head eye doth homage newappearing sight serving looks sacred majesty climbd steepup heavenly hill resembling strong youth middle age yet mortal looks adore beauty still attending golden pilgrimage highmost pitch weary car like feeble age reeleth day eyes fore duteous converted low tract look another way thou thyself outgoing thy noon unlookd diest unless thou get son
    70 tokens: music hear why hearst thou music sadly sweets sweets war joy delights joy why lovst thou thou receivst gladly else receivst pleasure thine annoy true concord welltuned sounds unions married offend thine ear sweetly chide thee confounds singleness parts thou shouldst bear mark string sweet husband another strikes mutual ordering resembling sire child happy mother pleasing note sing whose speechless song many seeming sings thee thou single wilt prove none
    70 tokens: fear wet widows eye thou consumst thy self single life ah thou issueless shalt hap die world wail thee like makeless wife world thy widow still weep thou form thee hast left behind every private widow well keep childrens eyes husbands shape mind look unthrift world doth spend shifts place still world enjoys beautys waste hath world end kept unused user destroys love toward others bosom sits murdrous shame commits
    69 tokens: shame deny thou bearst love thy self art unprovident grant thou wilt thou art belovd many thou none lovst evident thou art possessd murderous hate gainst thy self thou stickst conspire seeking beauteous roof ruinate repair thy chief desire o change thy thought change mind shall hate fairer lodgd gentle love thy presence gracious kind thyself least kindhearted prove make thee another self love beauty still live thine thee

단어 인코딩을 만듭니다.

enc = wordEncoding(documents)

enc = 
  wordEncoding with properties:

      NumWords: 3092
    Vocabulary: ["fairest"    "creatures"    "desire"    "increase"    "thereby"    "beautys"    "rose"    "might"    "never"    "die"    "riper"    "time"    "decease"    "tender"    "heir"    "bear"    "memory"    "thou"    ...    ] (1x3092 string)

단어 임베딩에서 단어 인코딩 만들기

단어 임베딩에서 단어 인코딩을 만들려면 단어 임베딩 단어집을 wordEncoding 함수에 단어 목록으로 입력합니다.

사전 훈련된 단어 임베딩을 불러옵니다.

emb = fastTextWordEmbedding;

단어집을 추출합니다.

words = emb.Vocabulary;

단어집을 사용하여 단어 인코딩을 만듭니다.

enc = wordEncoding(words)

enc = 
  wordEncoding with properties:

      NumWords: 999994
    Vocabulary: [1×999994 string]

단어 임베딩 가중치를 사용하여 딥러닝 신경망에서 해당 단어 임베딩 계층을 초기화하기 위해, word2vec 함수를 사용하여 계층 가중치를 추출하고 wordEmbeddingLayer 함수의 'Weights' 이름-값 쌍을 설정합니다. 단어 임베딩 계층에는 단어 벡터의 열이 필요하기 때문에 word2vec 함수의 출력값을 전치해야 합니다.

dimension = emb.Dimension;
numWords = numel(words);

layer = wordEmbeddingLayer(dimension,numWords, ...
    'Weights',word2vec(emb,words)')

layer = 
  WordEmbeddingLayer with properties:

         Name: ''

   Hyperparameters
    Dimension: 300
     NumWords: 999994

   Learnable Parameters
      Weights: [300×999994 single]

  Show all properties

문서 내 상위 단어의 단어 인코딩 만들기

예제 데이터를 불러옵니다. 파일 sonnetsPreprocessed.txt에는 셰익스피어 소네트의 전처리된 버전이 들어 있습니다. 파일에는 한 줄에 하나씩 소네트가 들어 있으며 단어가 공백으로 구분되어 있습니다. sonnetsPreprocessed.txt에서 텍스트를 추출하고, 추출한 텍스트를 새 줄 문자에서 문서로 분할한 후 그 문서를 토큰화합니다.

filename = "sonnetsPreprocessed.txt";
str = extractFileText(filename);
textData = split(str,newline);
documents = tokenizedDocument(textData);
documents(1:10)

ans = 
  10x1 tokenizedDocument:

    70 tokens: fairest creatures desire increase thereby beautys rose might never die riper time decease tender heir might bear memory thou contracted thine own bright eyes feedst thy lights flame selfsubstantial fuel making famine abundance lies thy self thy foe thy sweet self cruel thou art worlds fresh ornament herald gaudy spring thine own bud buriest thy content tender churl makst waste niggarding pity world else glutton eat worlds due grave thee
    71 tokens: forty winters shall besiege thy brow dig deep trenches thy beautys field thy youths proud livery gazed tatterd weed small worth held asked thy beauty lies treasure thy lusty days say thine own deep sunken eyes alleating shame thriftless praise praise deservd thy beautys thou couldst answer fair child mine shall sum count make old excuse proving beauty succession thine new made thou art old thy blood warm thou feelst cold
    65 tokens: look thy glass tell face thou viewest time face form another whose fresh repair thou renewest thou dost beguile world unbless mother fair whose uneard womb disdains tillage thy husbandry fond tomb selflove stop posterity thou art thy mothers glass thee calls back lovely april prime thou windows thine age shalt despite wrinkles thy golden time thou live rememberd die single thine image dies thee
    71 tokens: unthrifty loveliness why dost thou spend upon thy self thy beautys legacy natures bequest gives nothing doth lend frank lends free beauteous niggard why dost thou abuse bounteous largess thee give profitless usurer why dost thou great sum sums yet canst live traffic thy self alone thou thy self thy sweet self dost deceive nature calls thee gone acceptable audit canst thou leave thy unused beauty tombed thee lives th executor
    61 tokens: hours gentle work frame lovely gaze every eye doth dwell play tyrants same unfair fairly doth excel neverresting time leads summer hideous winter confounds sap checked frost lusty leaves quite gone beauty oersnowed bareness every summers distillation left liquid prisoner pent walls glass beautys effect beauty bereft nor nor remembrance flowers distilld though winter meet leese show substance still lives sweet
    68 tokens: let winters ragged hand deface thee thy summer ere thou distilld make sweet vial treasure thou place beautys treasure ere selfkilld forbidden usury happies pay willing loan thats thy self breed another thee ten times happier ten ten times thy self happier thou art ten thine ten times refigurd thee death thou shouldst depart leaving thee living posterity selfwilld thou art fair deaths conquest make worms thine heir
    64 tokens: lo orient gracious light lifts up burning head eye doth homage newappearing sight serving looks sacred majesty climbd steepup heavenly hill resembling strong youth middle age yet mortal looks adore beauty still attending golden pilgrimage highmost pitch weary car like feeble age reeleth day eyes fore duteous converted low tract look another way thou thyself outgoing thy noon unlookd diest unless thou get son
    70 tokens: music hear why hearst thou music sadly sweets sweets war joy delights joy why lovst thou thou receivst gladly else receivst pleasure thine annoy true concord welltuned sounds unions married offend thine ear sweetly chide thee confounds singleness parts thou shouldst bear mark string sweet husband another strikes mutual ordering resembling sire child happy mother pleasing note sing whose speechless song many seeming sings thee thou single wilt prove none
    70 tokens: fear wet widows eye thou consumst thy self single life ah thou issueless shalt hap die world wail thee like makeless wife world thy widow still weep thou form thee hast left behind every private widow well keep childrens eyes husbands shape mind look unthrift world doth spend shifts place still world enjoys beautys waste hath world end kept unused user destroys love toward others bosom sits murdrous shame commits
    69 tokens: shame deny thou bearst love thy self art unprovident grant thou wilt thou art belovd many thou none lovst evident thou art possessd murderous hate gainst thy self thou stickst conspire seeking beauteous roof ruinate repair thy chief desire o change thy thought change mind shall hate fairer lodgd gentle love thy presence gracious kind thyself least kindhearted prove make thee another self love beauty still live thine thee

단어 인코딩을 만듭니다. 빈도에 따라 인덱스를 정렬하고 상위 100개의 단어만 인코딩합니다.

enc = wordEncoding(documents, ...
    'Order','frequency', ...
    'MaxNumWords',100)

enc = 
  wordEncoding with properties:

      NumWords: 100
    Vocabulary: ["thy"    "thou"    "love"    "thee"    "doth"    "mine"    "shall"    "eyes"    "sweet"    "time"    "beauty"    "nor"    "art"    "yet"    "o"    "heart"    "thine"    "fair"    "make"    "hath"    "still"    ...    ] (1x100 string)

ind2word 함수를 사용하여 인덱스 1, 2, 3에 해당하는 단어를 표시합니다.

idx = [1 2 3];
words = ind2word(enc,idx)

words = 1x3 string
    "thy"    "thou"    "love"

인코딩 인덱스를 단어에 매핑하기

예제 데이터를 불러옵니다. 파일 sonnetsPreprocessed.txt에는 셰익스피어 소네트의 전처리된 버전이 들어 있습니다. 파일에는 한 줄에 하나씩 소네트가 들어 있으며 단어가 공백으로 구분되어 있습니다. sonnetsPreprocessed.txt에서 텍스트를 추출하고, 추출한 텍스트를 새 줄 문자에서 문서로 분할한 후 그 문서를 토큰화합니다.

filename = "sonnetsPreprocessed.txt";
str = extractFileText(filename);
textData = split(str,newline);
documents = tokenizedDocument(textData);
documents(1:10)

ans = 
  10x1 tokenizedDocument:

    70 tokens: fairest creatures desire increase thereby beautys rose might never die riper time decease tender heir might bear memory thou contracted thine own bright eyes feedst thy lights flame selfsubstantial fuel making famine abundance lies thy self thy foe thy sweet self cruel thou art worlds fresh ornament herald gaudy spring thine own bud buriest thy content tender churl makst waste niggarding pity world else glutton eat worlds due grave thee
    71 tokens: forty winters shall besiege thy brow dig deep trenches thy beautys field thy youths proud livery gazed tatterd weed small worth held asked thy beauty lies treasure thy lusty days say thine own deep sunken eyes alleating shame thriftless praise praise deservd thy beautys thou couldst answer fair child mine shall sum count make old excuse proving beauty succession thine new made thou art old thy blood warm thou feelst cold
    65 tokens: look thy glass tell face thou viewest time face form another whose fresh repair thou renewest thou dost beguile world unbless mother fair whose uneard womb disdains tillage thy husbandry fond tomb selflove stop posterity thou art thy mothers glass thee calls back lovely april prime thou windows thine age shalt despite wrinkles thy golden time thou live rememberd die single thine image dies thee
    71 tokens: unthrifty loveliness why dost thou spend upon thy self thy beautys legacy natures bequest gives nothing doth lend frank lends free beauteous niggard why dost thou abuse bounteous largess thee give profitless usurer why dost thou great sum sums yet canst live traffic thy self alone thou thy self thy sweet self dost deceive nature calls thee gone acceptable audit canst thou leave thy unused beauty tombed thee lives th executor
    61 tokens: hours gentle work frame lovely gaze every eye doth dwell play tyrants same unfair fairly doth excel neverresting time leads summer hideous winter confounds sap checked frost lusty leaves quite gone beauty oersnowed bareness every summers distillation left liquid prisoner pent walls glass beautys effect beauty bereft nor nor remembrance flowers distilld though winter meet leese show substance still lives sweet
    68 tokens: let winters ragged hand deface thee thy summer ere thou distilld make sweet vial treasure thou place beautys treasure ere selfkilld forbidden usury happies pay willing loan thats thy self breed another thee ten times happier ten ten times thy self happier thou art ten thine ten times refigurd thee death thou shouldst depart leaving thee living posterity selfwilld thou art fair deaths conquest make worms thine heir
    64 tokens: lo orient gracious light lifts up burning head eye doth homage newappearing sight serving looks sacred majesty climbd steepup heavenly hill resembling strong youth middle age yet mortal looks adore beauty still attending golden pilgrimage highmost pitch weary car like feeble age reeleth day eyes fore duteous converted low tract look another way thou thyself outgoing thy noon unlookd diest unless thou get son
    70 tokens: music hear why hearst thou music sadly sweets sweets war joy delights joy why lovst thou thou receivst gladly else receivst pleasure thine annoy true concord welltuned sounds unions married offend thine ear sweetly chide thee confounds singleness parts thou shouldst bear mark string sweet husband another strikes mutual ordering resembling sire child happy mother pleasing note sing whose speechless song many seeming sings thee thou single wilt prove none
    70 tokens: fear wet widows eye thou consumst thy self single life ah thou issueless shalt hap die world wail thee like makeless wife world thy widow still weep thou form thee hast left behind every private widow well keep childrens eyes husbands shape mind look unthrift world doth spend shifts place still world enjoys beautys waste hath world end kept unused user destroys love toward others bosom sits murdrous shame commits
    69 tokens: shame deny thou bearst love thy self art unprovident grant thou wilt thou art belovd many thou none lovst evident thou art possessd murderous hate gainst thy self thou stickst conspire seeking beauteous roof ruinate repair thy chief desire o change thy thought change mind shall hate fairer lodgd gentle love thy presence gracious kind thyself least kindhearted prove make thee another self love beauty still live thine thee

단어 인코딩을 만듭니다.

enc = wordEncoding(documents)

enc = 
  wordEncoding with properties:

      NumWords: 3092
    Vocabulary: ["fairest"    "creatures"    "desire"    "increase"    "thereby"    "beautys"    "rose"    "might"    "never"    "die"    "riper"    "time"    "decease"    "tender"    "heir"    "bear"    "memory"    "thou"    ...    ] (1x3092 string)

ind2word 함수를 사용하여 인덱스 1, 3, 5에 해당하는 단어를 표시합니다.

idx = [1 3 5];
words = ind2word(enc,idx)

words = 1x3 string
    "fairest"    "desire"    "thereby"

단어를 인코딩 인덱스에 매핑하기

예제 데이터를 불러옵니다. 파일 sonnetsPreprocessed.txt에는 셰익스피어 소네트의 전처리된 버전이 들어 있습니다. 파일에는 한 줄에 하나씩 소네트가 들어 있으며 단어가 공백으로 구분되어 있습니다. sonnetsPreprocessed.txt에서 텍스트를 추출하고, 추출한 텍스트를 새 줄 문자에서 문서로 분할한 후 그 문서를 토큰화합니다.

filename = "sonnetsPreprocessed.txt";
str = extractFileText(filename);
textData = split(str,newline);
documents = tokenizedDocument(textData);
documents(1:10)

ans = 
  10x1 tokenizedDocument:

    70 tokens: fairest creatures desire increase thereby beautys rose might never die riper time decease tender heir might bear memory thou contracted thine own bright eyes feedst thy lights flame selfsubstantial fuel making famine abundance lies thy self thy foe thy sweet self cruel thou art worlds fresh ornament herald gaudy spring thine own bud buriest thy content tender churl makst waste niggarding pity world else glutton eat worlds due grave thee
    71 tokens: forty winters shall besiege thy brow dig deep trenches thy beautys field thy youths proud livery gazed tatterd weed small worth held asked thy beauty lies treasure thy lusty days say thine own deep sunken eyes alleating shame thriftless praise praise deservd thy beautys thou couldst answer fair child mine shall sum count make old excuse proving beauty succession thine new made thou art old thy blood warm thou feelst cold
    65 tokens: look thy glass tell face thou viewest time face form another whose fresh repair thou renewest thou dost beguile world unbless mother fair whose uneard womb disdains tillage thy husbandry fond tomb selflove stop posterity thou art thy mothers glass thee calls back lovely april prime thou windows thine age shalt despite wrinkles thy golden time thou live rememberd die single thine image dies thee
    71 tokens: unthrifty loveliness why dost thou spend upon thy self thy beautys legacy natures bequest gives nothing doth lend frank lends free beauteous niggard why dost thou abuse bounteous largess thee give profitless usurer why dost thou great sum sums yet canst live traffic thy self alone thou thy self thy sweet self dost deceive nature calls thee gone acceptable audit canst thou leave thy unused beauty tombed thee lives th executor
    61 tokens: hours gentle work frame lovely gaze every eye doth dwell play tyrants same unfair fairly doth excel neverresting time leads summer hideous winter confounds sap checked frost lusty leaves quite gone beauty oersnowed bareness every summers distillation left liquid prisoner pent walls glass beautys effect beauty bereft nor nor remembrance flowers distilld though winter meet leese show substance still lives sweet
    68 tokens: let winters ragged hand deface thee thy summer ere thou distilld make sweet vial treasure thou place beautys treasure ere selfkilld forbidden usury happies pay willing loan thats thy self breed another thee ten times happier ten ten times thy self happier thou art ten thine ten times refigurd thee death thou shouldst depart leaving thee living posterity selfwilld thou art fair deaths conquest make worms thine heir
    64 tokens: lo orient gracious light lifts up burning head eye doth homage newappearing sight serving looks sacred majesty climbd steepup heavenly hill resembling strong youth middle age yet mortal looks adore beauty still attending golden pilgrimage highmost pitch weary car like feeble age reeleth day eyes fore duteous converted low tract look another way thou thyself outgoing thy noon unlookd diest unless thou get son
    70 tokens: music hear why hearst thou music sadly sweets sweets war joy delights joy why lovst thou thou receivst gladly else receivst pleasure thine annoy true concord welltuned sounds unions married offend thine ear sweetly chide thee confounds singleness parts thou shouldst bear mark string sweet husband another strikes mutual ordering resembling sire child happy mother pleasing note sing whose speechless song many seeming sings thee thou single wilt prove none
    70 tokens: fear wet widows eye thou consumst thy self single life ah thou issueless shalt hap die world wail thee like makeless wife world thy widow still weep thou form thee hast left behind every private widow well keep childrens eyes husbands shape mind look unthrift world doth spend shifts place still world enjoys beautys waste hath world end kept unused user destroys love toward others bosom sits murdrous shame commits
    69 tokens: shame deny thou bearst love thy self art unprovident grant thou wilt thou art belovd many thou none lovst evident thou art possessd murderous hate gainst thy self thou stickst conspire seeking beauteous roof ruinate repair thy chief desire o change thy thought change mind shall hate fairer lodgd gentle love thy presence gracious kind thyself least kindhearted prove make thee another self love beauty still live thine thee

단어 인코딩을 만듭니다.

enc = wordEncoding(documents)

enc = 
  wordEncoding with properties:

      NumWords: 3092
    Vocabulary: ["fairest"    "creatures"    "desire"    "increase"    "thereby"    "beautys"    "rose"    "might"    "never"    "die"    "riper"    "time"    "decease"    "tender"    "heir"    "bear"    "memory"    "thou"    ...    ] (1x3092 string)

word2ind 함수를 사용하여 단어 "rose", "love" 및 "beauty"를 인코딩 인덱스에 매핑합니다.

words = ["rose" "love" "beauty"];
idx = word2ind(enc,words)

idx = 1×3

     7   387    79

문서를 단어 인덱스 시퀀스로 변환하기

공장 보고서 데이터를 불러와서 tokenizedDocument 배열을 만듭니다.

filename = "factoryReports.csv";
data = readtable(filename,'TextType','string');
textData = data.Description;
documents = tokenizedDocument(textData);

단어 인코딩을 만듭니다.

enc = wordEncoding(documents);

문서를 단어 인덱스 시퀀스로 변환합니다.

sequences = doc2sequence(enc,documents);

처음 10개 시퀀스의 크기를 표시합니다. 각 시퀀스는 1xS 벡터입니다. 여기서 S는 시퀀스의 단어 인덱스 수입니다. 시퀀스가 채워지므로 S는 상수입니다.

sequences(1:10)

ans=10×1 cell array
    {[           0 0 0 0 0 0 0 1 2 3 4 5 6 7 8 9 10]}
    {[  0 0 0 0 0 0 11 12 13 14 15 2 16 17 18 19 10]}
    {[    0 0 0 0 0 0 20 2 21 22 7 23 24 25 7 26 10]}
    {[        0 0 0 0 0 0 0 0 0 0 0 27 28 6 7 18 10]}
    {[        0 0 0 0 0 0 0 0 0 0 0 0 29 30 7 31 10]}
    {[    0 0 0 0 0 0 0 32 33 6 7 34 35 36 37 38 10]}
    {[      0 0 0 0 0 0 0 0 0 39 40 36 41 6 7 42 10]}
    {[    0 0 0 0 0 0 0 0 43 44 22 45 46 47 7 48 10]}
    {[      0 0 0 0 0 0 0 0 0 0 49 50 17 7 51 48 10]}
    {[0 0 0 0 52 8 53 36 54 55 56 57 58 59 22 60 10]}

버전 내역

R2018b에 개발됨

참고 항목

fastTextWordEmbedding | doc2sequence | wordEmbeddingLayer | word2ind | ind2word | isVocabularyWord | wordEmbedding | tokenizedDocument | word2vec

도움말 항목

• 감성 분류기 훈련시키기
• 딥러닝을 사용하여 텍스트 데이터 분류하기
• 텍스트 산점도 플롯을 사용하여 단어 임베딩 시각화하기
• 분석할 텍스트 데이터 준비하기