旅游兼职网站建设,信阳企业网站建设,会计培训班初级费用,python的网站开发源码目录
1 keras实现Deepfm
demo
2 deepctr模版
3 其他实现方式 ctr_Kera 模型 数据集 预处理 执行步骤
4何为focal loss
参考 1 keras实现Deepfm 假设我们有两种 field 的特征#xff0c;连续型和离散型#xff0c;连续型 field 一般不做处理沿用原值#xff0c;离散型一…目录
1 keras实现Deepfm
demo
2 deepctr模版
3 其他实现方式 ctr_Kera 模型 数据集 预处理 执行步骤
4何为focal loss
参考 1 keras实现Deepfm 假设我们有两种 field 的特征连续型和离散型连续型 field 一般不做处理沿用原值离散型一般会做One-hot编码。离散型又能进一步分为单值型和多值型单值型在Onehot后的稀疏向量中只有一个特征为1其余都是0而多值型在Onehot后有多于1个特征为1其余是0。
DeepFM模型本质 将Wide Deep 部分的wide部分由 人工特征工程LR 转换为FM模型避开了人工特征工程 FM模型与deep part共享feature embedding。
为什么要用FM代替线性部分wide呢
因为线性模型有个致命的缺点无法提取高阶的组合特征。 FM通过隐向量latent vector做内积来表示组合特征从理论上解决了低阶和高阶组合特征提取的问题。但是实际应用中受限于计算复杂度一般也就只考虑到2阶交叉特征。
DeepFM包含两部分神经网络部分与因子分解机部分分别负责低阶特征的提取和高阶特征的提取。这两部分共享同样的输入。
不需要预训练 FM 得到隐向量 不需要人工特征工程 能同时学习低阶和高阶的组合特征 FM 模块和 Deep 模块共享 Feature Embedding 部分可以更快的训练以及更精确的训练学习。
https://blog.csdn.net/m0_51933492/article/details/126888136 # FM的一阶特征运算
#将预处理完的类别特征列合并
cat_columns [movie_ind_col, user_ind_col, user_genre_ind_col, item_genre_ind_col]#数值型特征不用经过独热编码和特征稠密化转换成可以输入神经网络的dense类型直接输入网络【推荐系统】DeepFM模型_deepfm多分类求auc_—Xi—的博客-CSDN博客 Model: model
__________________________________________________________________________________________________
Layer (type) Output Shape Param # Connected to movieAvgRating (InputLayer) [(None,)] 0
__________________________________________________________________________________________________
movieGenre1 (InputLayer) [(None,)] 0
__________________________________________________________________________________________________
movieGenre2 (InputLayer) [(None,)] 0
__________________________________________________________________________________________________
movieGenre3 (InputLayer) [(None,)] 0
__________________________________________________________________________________________________
movieId (InputLayer) [(None,)] 0
__________________________________________________________________________________________________
movieRatingCount (InputLayer) [(None,)] 0
__________________________________________________________________________________________________
movieRatingStddev (InputLayer) [(None,)] 0
__________________________________________________________________________________________________
releaseYear (InputLayer) [(None,)] 0
__________________________________________________________________________________________________
userAvgRating (InputLayer) [(None,)] 0
__________________________________________________________________________________________________
userGenre1 (InputLayer) [(None,)] 0
__________________________________________________________________________________________________
userGenre2 (InputLayer) [(None,)] 0
__________________________________________________________________________________________________
userGenre3 (InputLayer) [(None,)] 0
__________________________________________________________________________________________________
userGenre4 (InputLayer) [(None,)] 0
__________________________________________________________________________________________________
userGenre5 (InputLayer) [(None,)] 0
__________________________________________________________________________________________________
userId (InputLayer) [(None,)] 0
__________________________________________________________________________________________________
userRatedMovie1 (InputLayer) [(None,)] 0
__________________________________________________________________________________________________
userRatingCount (InputLayer) [(None,)] 0
__________________________________________________________________________________________________
userRatingStddev (InputLayer) [(None,)] 0
__________________________________________________________________________________________________
dense_features_16 (DenseFeature (None, 10) 190 movieAvgRating[0][0] movieGenre1[0][0] movieGenre2[0][0] movieGenre3[0][0] movieId[0][0] movieRatingCount[0][0] movieRatingStddev[0][0] releaseYear[0][0] userAvgRating[0][0] userGenre1[0][0] userGenre2[0][0] userGenre3[0][0] userGenre4[0][0] userGenre5[0][0] userId[0][0] userRatedMovie1[0][0] userRatingCount[0][0] userRatingStddev[0][0]
__________________________________________________________________________________________________
dense_features_17 (DenseFeature (None, 10) 10010 movieAvgRating[0][0] movieGenre1[0][0] movieGenre2[0][0] movieGenre3[0][0] movieId[0][0] movieRatingCount[0][0] movieRatingStddev[0][0] releaseYear[0][0] userAvgRating[0][0] userGenre1[0][0] userGenre2[0][0] userGenre3[0][0] userGenre4[0][0] userGenre5[0][0] userId[0][0] userRatedMovie1[0][0] userRatingCount[0][0] userRatingStddev[0][0]
__________________________________________________________________________________________________
dense_features_18 (DenseFeature (None, 10) 190 movieAvgRating[0][0] movieGenre1[0][0] movieGenre2[0][0] movieGenre3[0][0] movieId[0][0] movieRatingCount[0][0] movieRatingStddev[0][0] releaseYear[0][0] userAvgRating[0][0] userGenre1[0][0] userGenre2[0][0] userGenre3[0][0] userGenre4[0][0] userGenre5[0][0] userId[0][0] userRatedMovie1[0][0] userRatingCount[0][0] userRatingStddev[0][0]
__________________________________________________________________________________________________
dense_features_19 (DenseFeature (None, 10) 300010 movieAvgRating[0][0] movieGenre1[0][0] movieGenre2[0][0] movieGenre3[0][0] movieId[0][0] movieRatingCount[0][0] movieRatingStddev[0][0] releaseYear[0][0] userAvgRating[0][0] userGenre1[0][0] userGenre2[0][0] userGenre3[0][0] userGenre4[0][0] userGenre5[0][0] userId[0][0] userRatedMovie1[0][0] userRatingCount[0][0] userRatingStddev[0][0]
__________________________________________________________________________________________________
dense_features_20 (DenseFeature (None, 7) 0 movieAvgRating[0][0] movieGenre1[0][0] movieGenre2[0][0] movieGenre3[0][0] movieId[0][0] movieRatingCount[0][0] movieRatingStddev[0][0] releaseYear[0][0] userAvgRating[0][0] userGenre1[0][0] userGenre2[0][0] userGenre3[0][0] userGenre4[0][0] userGenre5[0][0] userId[0][0] userRatedMovie1[0][0] userRatingCount[0][0] userRatingStddev[0][0]
__________________________________________________________________________________________________
dense_18 (Dense) (None, 64) 704 dense_features_16[0][0]
__________________________________________________________________________________________________
dense_19 (Dense) (None, 64) 704 dense_features_17[0][0]
__________________________________________________________________________________________________
dense_20 (Dense) (None, 64) 704 dense_features_18[0][0]
__________________________________________________________________________________________________
dense_21 (Dense) (None, 64) 704 dense_features_19[0][0]
__________________________________________________________________________________________________
dense_22 (Dense) (None, 64) 512 dense_features_20[0][0]
__________________________________________________________________________________________________
reshape_10 (Reshape) (None, 10, 64) 0 dense_18[0][0]
__________________________________________________________________________________________________
reshape_11 (Reshape) (None, 10, 64) 0 dense_19[0][0]
__________________________________________________________________________________________________
reshape_12 (Reshape) (None, 10, 64) 0 dense_20[0][0]
__________________________________________________________________________________________________
reshape_13 (Reshape) (None, 10, 64) 0 dense_21[0][0]
__________________________________________________________________________________________________
reshape_14 (Reshape) (None, 10, 64) 0 dense_22[0][0]
__________________________________________________________________________________________________
concatenate_2 (Concatenate) (None, 50, 64) 0 reshape_10[0][0] reshape_11[0][0] reshape_12[0][0] reshape_13[0][0] reshape_14[0][0]
__________________________________________________________________________________________________
dense_features_14 (DenseFeature (None, 31040) 0 movieAvgRating[0][0] movieGenre1[0][0] movieGenre2[0][0] movieGenre3[0][0] movieId[0][0] movieRatingCount[0][0] movieRatingStddev[0][0] releaseYear[0][0] userAvgRating[0][0] userGenre1[0][0] userGenre2[0][0] userGenre3[0][0] userGenre4[0][0] userGenre5[0][0] userId[0][0] userRatedMovie1[0][0] userRatingCount[0][0] userRatingStddev[0][0]
__________________________________________________________________________________________________
dense_features_15 (DenseFeature (None, 7) 0 movieAvgRating[0][0] movieGenre1[0][0] movieGenre2[0][0] movieGenre3[0][0] movieId[0][0] movieRatingCount[0][0] movieRatingStddev[0][0] releaseYear[0][0] userAvgRating[0][0] userGenre1[0][0] userGenre2[0][0] userGenre3[0][0] userGenre4[0][0] userGenre5[0][0] userId[0][0] userRatedMovie1[0][0] userRatingCount[0][0] userRatingStddev[0][0]
__________________________________________________________________________________________________
reduce_layer (ReduceLayer) (None, 64) 0 concatenate_2[0][0]
__________________________________________________________________________________________________
multiply_1 (Multiply) (None, 50, 64) 0 concatenate_2[0][0] concatenate_2[0][0]
__________________________________________________________________________________________________
flatten_2 (Flatten) (None, 3200) 0 concatenate_2[0][0]
__________________________________________________________________________________________________
dense_16 (Dense) (None, 1) 31041 dense_features_14[0][0]
__________________________________________________________________________________________________
dense_17 (Dense) (None, 1) 8 dense_features_15[0][0]
__________________________________________________________________________________________________
multiply (Multiply) (None, 64) 0 reduce_layer[0][0] reduce_layer[0][0]
__________________________________________________________________________________________________
reduce_layer_1 (ReduceLayer) (None, 64) 0 multiply_1[0][0]
__________________________________________________________________________________________________
dense_23 (Dense) (None, 32) 102432 flatten_2[0][0]
__________________________________________________________________________________________________
add_2 (Add) (None, 1) 0 dense_16[0][0] dense_17[0][0]
__________________________________________________________________________________________________
subtract (Subtract) (None, 64) 0 multiply[0][0] reduce_layer_1[0][0]
__________________________________________________________________________________________________
dense_24 (Dense) (None, 16) 528 dense_23[0][0]
__________________________________________________________________________________________________
concatenate_3 (Concatenate) (None, 81) 0 add_2[0][0] subtract[0][0] dense_24[0][0]
__________________________________________________________________________________________________
dense_25 (Dense) (None, 1) 82 concatenate_3[0][0] Total params: 447,819
Trainable params: 447,819
Non-trainable params: 0
__________________________________________________________________________________________________
None TensorFlow的plot_model功能_plot_model函数_buptwhq的博客-CSDN博客
from keras.utils.vis_utils import plot_model ... plot_model(model, to_filemodel.png, show_shapesTrue, show_layer_namesFalse, rankdirTB)8. keras - 绘制网络结构_keras 设计网络_Micheal超的博客-CSDN博客
问题一什么是特征交互为什么要进行特征交互
二阶特征交互通过对主流应用市场的研究我们发现人们经常在用餐时间下载送餐的应用程序这就表明应用类别和时间戳之间的阶数-2交互作用是CTR预测的一个信号。 三阶或者高阶特征交互我们还发现男性青少年喜欢射击游戏和RPG游戏这意味着应用类别、用户性别和年龄的(阶数-3)交互是CTR的另一个信号。 根据谷歌的WD模型的应用 作者发现同时考虑低阶和高阶的交互特征比单独考虑其中之一有更多的改进 问题二为啥人工特征工程有挑战性
一些特征工程比较容易理解就比如上面提到的那两个 这时候往往我们都能很容易的设计或者组合那样的特征。 然而其他大部分特征交互都隐藏在数据中难以先验识别比如经典的关联规则 尿布和啤酒 就是从数据中挖掘出来的而不是由专家发现的只能由机器学习自动捕捉即使是对于容易理解的交互专家们似乎也不可能详尽地对它们进行建模特别是当特征的数量很大的时候.。 demo
【CTR模型】TensorFlow2.0 的 xDeepFM 实现与实战(附代码数据)_tensorflow ctr 模型_VariableX的博客-CSDN博客 python相减substract_浅谈keras中的Merge层(实现层的相加、相减、相乘实例)_林祈墨的博客-CSDN博客
# coding:utf-8
from keras.layers import *
from keras.models import Model
from MyMeanPooling import MyMeanPool
from MySumLayer import MySumLayer
from MyFlatten import MyFlatten
from keras.utils import plot_model# numeric fields
in_score Input(shape[1], namescore) # None*1
in_sales Input(shape[1], namesales) # None*1
# single value categorical fields
in_gender Input(shape[1], namegender) # None*1
in_age Input(shape[1], nameage) # None*1
# multiple value categorical fields
in_interest Input(shape[3], nameinterest) # None*3, 最长长度3
in_topic Input(shape[4], nametopic) # None*4, 最长长度4First Order Embeddings
numeric Concatenate()([in_score, in_sales]) # None*2
dense_numeric Dense(1)(numeric) # None*1
emb_gender_1d Reshape([1])(Embedding(3, 1)(in_gender)) # None*1, 性别取值3种
emb_age_1d Reshape([1])(Embedding(10, 1)(in_age)) # None*1, 年龄取值10种
emb_interest_1d Embedding(11, 1, mask_zeroTrue)(in_interest) # None*3*1
emb_interest_1d MyMeanPool(axis1)(emb_interest_1d) # None*1
emb_topic_1d Embedding(22, 1, mask_zeroTrue)(in_topic) # None*4*1
emb_topic_1d MyMeanPool(axis1)(emb_topic_1d) # None*1compute
y_first_order Add()([dense_numeric,emb_gender_1d, emb_age_1d,emb_interest_1d,emb_topic_1d]) # None*1latent 8
Second Order Embeddings
emb_score_Kd RepeatVector(1)(Dense(latent)(in_score)) # None * 1 * K
emb_sales_Kd RepeatVector(1)(Dense(latent)(in_sales)) # None * 1 * K
emb_gender_Kd Embedding(3, latent)(in_gender) # None * 1 * K
emb_age_Kd Embedding(10, latent)(in_age) # None * 1 * K
emb_interest_Kd Embedding(11, latent, mask_zeroTrue)(in_interest) # None * 3 * K
emb_interest_Kd RepeatVector(1)(MyMeanPool(axis1)(emb_interest_Kd)) # None * 1 * K
emb_topic_Kd Embedding(22, latent, mask_zeroTrue)(in_topic) # None * 4 * K
emb_topic_Kd RepeatVector(1)(MyMeanPool(axis1)(emb_topic_Kd)) # None * 1 * Kemb Concatenate(axis1)([emb_score_Kd,emb_sales_Kd,emb_gender_Kd,emb_age_Kd,emb_interest_Kd,emb_topic_Kd]) # None * 6 * Kcompute
summed_features_emb MySumLayer(axis1)(emb) # None * K
summed_features_emb_square Multiply()([summed_features_emb,summed_features_emb]) # None * Ksquared_features_emb Multiply()([emb, emb]) # None * 9 * K
squared_sum_features_emb MySumLayer(axis1)(squared_features_emb) # Non * Ksub Subtract()([summed_features_emb_square, squared_sum_features_emb]) # None * K
sub Lambda(lambda x:x*0.5)(sub) # None * Ky_second_order MySumLayer(axis1)(sub) # None * 1deep parts
y_deep MyFlatten()(emb) # None*(6*K)
y_deep Dropout(0.5)(Dense(128, activationrelu)(y_deep))
y_deep Dropout(0.5)(Dense(64, activationrelu)(y_deep))
y_deep Dropout(0.5)(Dense(32, activationrelu)(y_deep))
y_deep Dropout(0.5)(Dense(1, activationrelu)(y_deep))deepFM
y Concatenate(axis1)([y_first_order, y_second_order,y_deep])
y Dense(1, activationsigmoid)(y)model Model(inputs[in_score, in_sales,in_gender, in_age,in_interest, in_topic],outputs[y])plot_model(model, model.png, show_shapesTrue)
in_interest本身one-hot 2 deepctr模版
DeepCTR DeepMatch简单实用指南_tf.Print(**)的博客-CSDN博客 文档
Welcome to DeepCTR’s documentation! — DeepCTR 0.9.3 documentation
模型保存
FAQ — DeepCTR 0.9.3 documentation
deepctr.models.deepfm module — DeepCTR 0.9.3 documentation
model DeepFM()
model.save_weights(DeepFM_w.h5)
model.load_weights(DeepFM_w.h5)
sparse_features [C str(i) for i in range(1, 27)]
#稀疏离散特征
dense_features [I str(i) for i in range(1, 14)]
#连续全连接
def get_feature_names(feature_columns):features build_input_features(feature_columns)return list(features.keys())def build_input_features(feature_columns, prefix):input_features OrderedDict()# 字典for fc in feature_columns:if isinstance(fc, SparseFeat):input_features[fc.name] Input(shape(1,), nameprefix fc.name, dtypefc.dtype)elif isinstance(fc, DenseFeat):input_features[fc.name] Input(shape(fc.dimension,), nameprefix fc.name, dtypefc.dtype)elif isinstance(fc, VarLenSparseFeat):input_features[fc.name] Input(shape(fc.maxlen,), nameprefix fc.name,dtypefc.dtype)if fc.weight_name is not None:input_features[fc.weight_name] Input(shape(fc.maxlen, 1), nameprefix fc.weight_name,dtypefloat32)if fc.length_name is not None:input_features[fc.length_name] Input((1,), nameprefix fc.length_name, dtypeint32)else:raise TypeError(Invalid feature column type,got, type(fc))return input_features
class DenseFeat(namedtuple(DenseFeat, [name, dimension, dtype, transform_fn])): Dense featureArgs:name: feature name.dimension: dimension of the feature, default 1.dtype: dtype of the feature, defaultfloat32.transform_fn: If not None , a function that can be used to transformvalues of the feature. the function takes the input Tensor as itsargument, and returns the output Tensor.(e.g. lambda x: (x - 3.0) / 4.2).__slots__ ()def __new__(cls, name, dimension1, dtypefloat32, transform_fnNone):return super(DenseFeat, cls).__new__(cls, name, dimension, dtype, transform_fn)def __hash__(self):return self.name.__hash__()# def __eq__(self, other):# if self.name other.name:# return True# return False# def __repr__(self):# return DenseFeat:self.name
class SparseFeat(namedtuple(SparseFeat,[name, vocabulary_size, embedding_dim, use_hash, vocabulary_path, dtype, embeddings_initializer,embedding_name,group_name, trainable])):__slots__ ()def __new__(cls, name, vocabulary_size, embedding_dim4, use_hashFalse, vocabulary_pathNone, dtypeint32, embeddings_initializerNone,embedding_nameNone,group_nameDEFAULT_GROUP_NAME, trainableTrue):if embedding_dim auto:embedding_dim 6 * int(pow(vocabulary_size, 0.25))if embeddings_initializer is None:embeddings_initializer RandomNormal(mean0.0, stddev0.0001, seed2020)if embedding_name is None:embedding_name namereturn super(SparseFeat, cls).__new__(cls, name, vocabulary_size, embedding_dim, use_hash, vocabulary_path, dtype,embeddings_initializer,embedding_name, group_name, trainable)
使用例子
python:sklearn标签编码(LabelEncoder)_sklearn labelencoder_jenny_paofu的博客-CSDN博客
from sklearn import preprocessingdata[电脑,手机,手机,手表]encpreprocessing.LabelEncoder()
encenc.fit([电脑,手机,手表])#训练LabelEncoder,将电脑手表手机编码为0,1,2
dataenc.transform(data)#使用训练好的LabelEncoder对原数据进行编码也叫归一化print(data)#[2 0 0 1]import pandas as pd
from sklearn.metrics import log_loss, roc_auc_score
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import LabelEncoder, MinMaxScalerfrom deepctr.models import DeepFM
from deepctr.feature_column import SparseFeat, DenseFeat, get_feature_namesif __name__ __main__:data pd.read_csv(./criteo_sample.txt)sparse_features [C str(i) for i in range(1, 27)]dense_features [I str(i) for i in range(1, 14)]data[sparse_features] data[sparse_features].fillna(-1, )data[dense_features] data[dense_features].fillna(0, )target [label]# 1.Label Encoding for sparse features,and do simple Transformation for dense featuresfor feat in sparse_features:lbe LabelEncoder()data[feat] lbe.fit_transform(data[feat])mms MinMaxScaler(feature_range(0, 1))data[dense_features] mms.fit_transform(data[dense_features])# 2.count #unique features for each sparse field,and record dense feature field namefixlen_feature_columns [SparseFeat(feat, vocabulary_sizedata[feat].nunique(),embedding_dim4 )for i,feat in enumerate(sparse_features)] [DenseFeat(feat, 1,)for feat in dense_features]dnn_feature_columns fixlen_feature_columnslinear_feature_columns fixlen_feature_columnsfeature_names get_feature_names(linear_feature_columns dnn_feature_columns)# 3.generate input data for modeltrain, test train_test_split(data, test_size0.2, random_state2018)train_model_input {name:train[name] for name in feature_names}test_model_input {name:test[name] for name in feature_names}# 4.Define Model,train,predict and evaluatemodel DeepFM(linear_feature_columns, dnn_feature_columns, taskbinary)model.compile(adam, binary_crossentropy,metrics[binary_crossentropy], )history model.fit(train_model_input, train[target].values,batch_size256, epochs10, verbose2, validation_split0.2, )pred_ans model.predict(test_model_input, batch_size256)print(test LogLoss, round(log_loss(test[target].values, pred_ans), 4))print(test AUC, round(roc_auc_score(test[target].values, pred_ans), 4))
fm
输入 model Model(inputsinputs_list, outputsoutput)return model 输入样式
def input_from_feature_columns(features, feature_columns, l2_reg, seed, prefix, seq_mask_zeroTrue,support_denseTrue, support_groupFalse):sparse_feature_columns list(filter(lambda x: isinstance(x, SparseFeat), feature_columns)) if feature_columns else []varlen_sparse_feature_columns list(filter(lambda x: isinstance(x, VarLenSparseFeat), feature_columns)) if feature_columns else []embedding_matrix_dict create_embedding_matrix(feature_columns, l2_reg, seed, prefixprefix,seq_mask_zeroseq_mask_zero)group_sparse_embedding_dict embedding_lookup(embedding_matrix_dict, features, sparse_feature_columns)dense_value_list get_dense_input(features, feature_columns)if not support_dense and len(dense_value_list) 0:raise ValueError(DenseFeat is not supported in dnn_feature_columns)sequence_embed_dict varlen_embedding_lookup(embedding_matrix_dict, features, varlen_sparse_feature_columns)group_varlen_sparse_embedding_dict get_varlen_pooling_list(sequence_embed_dict, features,varlen_sparse_feature_columns)group_embedding_dict mergeDict(group_sparse_embedding_dict, group_varlen_sparse_embedding_dict)if not support_group:group_embedding_dict list(chain.from_iterable(group_embedding_dict.values()))return group_embedding_dict, dense_value_listdef build_input_features(feature_columns, prefix):input_features OrderedDict()for fc in feature_columns:if isinstance(fc, SparseFeat):input_features[fc.name] Input(shape(1,), nameprefix fc.name, dtypefc.dtype)elif isinstance(fc, DenseFeat):input_features[fc.name] Input(shape(fc.dimension,), nameprefix fc.name, dtypefc.dtype)elif isinstance(fc, VarLenSparseFeat):input_features[fc.name] Input(shape(fc.maxlen,), nameprefix fc.name,dtypefc.dtype)if fc.weight_name is not None:input_features[fc.weight_name] Input(shape(fc.maxlen, 1), nameprefix fc.weight_name,dtypefloat32)if fc.length_name is not None:input_features[fc.length_name] Input((1,), nameprefix fc.length_name, dtypeint32)else:raise TypeError(Invalid feature column type,got, type(fc))return
embedding
深度学习框架Keras中的embedding简单理解 - 简书
def create_embedding_dict(sparse_feature_columns, varlen_sparse_feature_columns, seed, l2_reg,prefixsparse_, seq_mask_zeroTrue):sparse_embedding {}for feat in sparse_feature_columns:emb Embedding(feat.vocabulary_size, feat.embedding_dim,embeddings_initializerfeat.embeddings_initializer,embeddings_regularizerl2(l2_reg),nameprefix _emb_ feat.embedding_name)emb.trainable feat.trainablesparse_embedding[feat.embedding_name] embif varlen_sparse_feature_columns and len(varlen_sparse_feature_columns) 0:for feat in varlen_sparse_feature_columns:# if feat.name not in sparse_embedding:emb Embedding(feat.vocabulary_size, feat.embedding_dim,embeddings_initializerfeat.embeddings_initializer,embeddings_regularizerl2(l2_reg),nameprefix _seq_emb_ feat.name,mask_zeroseq_mask_zero)emb.trainable feat.trainablesparse_embedding[feat.embedding_name] embreturn
# -*- coding:utf-8 -*-Author:Weichen Shen, weichenswc163.com
Reference:[1] Guo H, Tang R, Ye Y, et al. Deepfm: a factorization-machine based neural network for ctr prediction[J]. arXiv preprint arXiv:1703.04247, 2017.(https://arxiv.org/abs/1703.04247)
from itertools import chainfrom tensorflow.python.keras.models import Model
from tensorflow.python.keras.layers import Densefrom ..feature_column import build_input_features, get_linear_logit, DEFAULT_GROUP_NAME, input_from_feature_columns
from ..layers.core import PredictionLayer, DNN
from ..layers.interaction import FM
from ..layers.utils import concat_func, add_func, combined_dnn_inputdef DeepFM(linear_feature_columns, dnn_feature_columns, fm_group(DEFAULT_GROUP_NAME,), dnn_hidden_units(256, 128, 64),l2_reg_linear0.00001, l2_reg_embedding0.00001, l2_reg_dnn0, seed1024, dnn_dropout0,dnn_activationrelu, dnn_use_bnFalse, taskbinary):Instantiates the DeepFM Network architecture.:param linear_feature_columns: An iterable containing all the features used by the linear part of the model.:param dnn_feature_columns: An iterable containing all the features used by the deep part of the model.:param fm_group: list, group_name of features that will be used to do feature interactions.:param dnn_hidden_units: list,list of positive integer or empty list, the layer number and units in each layer of DNN:param l2_reg_linear: float. L2 regularizer strength applied to linear part:param l2_reg_embedding: float. L2 regularizer strength applied to embedding vector:param l2_reg_dnn: float. L2 regularizer strength applied to DNN:param seed: integer ,to use as random seed.:param dnn_dropout: float in [0,1), the probability we will drop out a given DNN coordinate.:param dnn_activation: Activation function to use in DNN:param dnn_use_bn: bool. Whether use BatchNormalization before activation or not in DNN:param task: str, binary for binary logloss or regression for regression loss:return: A Keras model instance.features build_input_features(linear_feature_columns dnn_feature_columns)inputs_list list(features.values())linear_logit get_linear_logit(features, linear_feature_columns, seedseed, prefixlinear,l2_regl2_reg_linear)group_embedding_dict, dense_value_list input_from_feature_columns(features, dnn_feature_columns, l2_reg_embedding,seed, support_groupTrue)fm_logit add_func([FM()(concat_func(v, axis1))for k, v in group_embedding_dict.items() if k in fm_group])dnn_input combined_dnn_input(list(chain.from_iterable(group_embedding_dict.values())), dense_value_list)dnn_output DNN(dnn_hidden_units, dnn_activation, l2_reg_dnn, dnn_dropout, dnn_use_bn, seedseed)(dnn_input)dnn_logit Dense(1, use_biasFalse)(dnn_output)final_logit add_func([linear_logit, fm_logit, dnn_logit])output PredictionLayer(task)(final_logit)model Model(inputsinputs_list, outputsoutput)return model3 其他实现方式
GitHub - as472780551/ctr_Keras: LR, WideDeep, DCN, NFM, DeepFM, NFFM
ctr_Keras
很简单的ctr模型实现欢迎指出bug提出宝贵意见
模型
LR
FNNhttp://www0.cs.ucl.ac.uk/staff/w.zhang/rtb-papers/deep-ctr.pdf
WideDeephttps://arxiv.org/abs/1606.07792
IPNNhttps://arxiv.org/abs/1611.00144
DCNhttps://arxiv.org/abs/1708.05123
NFMhttps://www.comp.nus.edu.sg/~xiangnan/papers/sigir17-nfm.pdf
DeepFMhttps://arxiv.org/abs/1703.04247
NFFM腾讯赛冠军模型
数据集
kaggle-criteo-2014 dataset
Kaggle Display Advertising Challenge Dataset - Criteo Engineering
数据集按9:1划分请自行划分
traintest412565564584063
预处理
连续型特征(13)缺失值补0离散分桶
离散型特征(26)过滤频率低于10的特征值
执行步骤
运行preprocess.py生成train.csv和test.csv文件
python preprocess.py运行相应的ctr模型代码文件如
python lr.py
代码
Embedding理解及keras中Embedding参数详解代码案例说明_小时不识月123的博客-CSDN博客
from keras.layers.embeddings import Embedding
keras.layers.Embedding(input_dim, #词汇表大小就是你的文本里你感兴趣词的数量
output_dim, #词向量的维度
embeddings_initializeruniform,# Embedding矩阵的初始化方法
embeddings_regularizerNone,# Embedding matrix 的正则化方法
activity_regularizerNone,
embeddings_constraintNone, # Embedding matrix 的约束函数
mask_zeroFalse, #是否把 0 看作padding 值取值为True时接下来的所有层都必须支持 masking词汇表的索引要从1开始因为文档填充用的是0如果词汇表索引从0开始会产生混淆input_dim
vocabulary 1
input_lengthNone)# 输入序列的长度就是文档经过padding后的向量的长度。函数输入尺寸为batch_size, input_length的2D张量
batch_size就是你的mini batch里的样本量
input_length就是你的文档转化成索引向量每个词用词索引表示的向量后的维数。函数输出尺寸为batch_size, input_length,output_dim的3D张量
上面说了output_dim就是词向量的维度就是词转化为向量这个向量的维度
比如word2vec把“哈哈”转化为向量[1.01,2.9,3]那么output_dim就是3.embedding
#!/usr/bin/python
# -*- coding: utf-8 -*-author:
contact:
time:
context:from keras.layers.embeddings import Embedding
from keras.models import Sequential
import numpy as np#我们随机生成第一层输入即每个样本存储于单独的list此list里的每个特征或者说元素用正整数索引表示同时所有样本构成list
input_array np.random.randint(1000, size(32, 10))[[250 219 228 56 572 110 467 214 173 342][678 13 994 406 678 995 966 398 732 715]...[426 519 254 180 235 707 887 962 834 269][775 380 706 784 842 369 514 265 797 976][666 832 821 953 369 836 656 808 562 263]]
model Sequential()
model.add(Embedding(1000, 64, input_length10))#词汇表里词999,词向量的维度64,输入序列的长度10
# keras.layers.Embedding(input_dim, output_dim, input_length)#词汇表大小,词向量的维度,输入序列的长度print(model.input_shape)
print(model.output_shape)(None, 10) #其中 None的取值是batch_size
(None, 10, 64)input_shape:函数输入,尺寸为batch_size, 10的2D张量(矩阵的意思)
output_shape:函数输出,尺寸为batch_size, 10,64的3D张量
model.compile(rmsprop, mse)
output_array model.predict(input_array)
assert output_array.shape (32, 10, 64)
print(output_array)
print(len(output_array))
print(len(output_array[1]))
print(len(output_array[1][1]))[
[[] [] [] ... [] [] []]
[[] [] [] ... [] [] []]
...
[[] [] [] ... [] [] []]
]32:最外层维数3232个样本
10第二层维数10每个样本用10个词表示
64最内层维数64每个词用64维向量表示for cat in cat_columns:input Input(shape(1,))cat_field_input.append(input)nums pd.concat((train[cat], test[cat])).max() 1embed Embedding(nums, 1, input_length1, trainableTrue)(input)
def base_model(cat_columns, train, test):cat_num len(cat_columns)field_cnt cat_numcat_field_input []field_embedding []lr_embedding []for cat in cat_columns:input Input(shape(1,))cat_field_input.append(input)nums pd.concat((train[cat], test[cat])).max() 1embed Embedding(nums, 1, input_length1, trainableTrue)(input)reshape Reshape((1,))(embed)lr_embedding.append(reshape)# fm embeddingsfield []embed Embedding(nums, 10, input_length1, trainableTrue)(input)reshape Reshape((10,))(embed)field_embedding.append(reshape)# fm embeddings#######fm layer##########fm_square Lambda(lambda x: K.square(x))(add(field_embedding))square_fm add([Lambda(lambda x:K.square(x))(embed)for embed in field_embedding])inner_product subtract([fm_square, square_fm])inner_product Lambda(lambda x: K.sum(x / 2, axis-1, keepdimsTrue))(inner_product)#######dnn layer##########embed_layer concatenate(field_embedding, axis-1)embed_layer Dense(64)(embed_layer)embed_layer BatchNormalization()(embed_layer)embed_layer Activation(relu)(embed_layer)embed_layer Dense(64)(embed_layer)embed_layer BatchNormalization()(embed_layer)embed_layer Activation(relu)(embed_layer)embed_layer Dense(1)(embed_layer)########linear layer##########lr_layer add(lr_embedding [embed_layer, inner_product])preds Activation(sigmoid)(lr_layer)opt Adam(0.001)model Model(inputscat_field_input, outputspreds)model.compile(lossbinary_crossentropy, optimizeropt, metrics[acc])print(model.summary())return model
4何为focal loss
focal loss 是随网络RetinaNet一起提出的一个令人惊艳的损失函数 paper 下载主要针对的是解决正负样本比例严重偏斜所产生的模型难以训练的问题。 在keras中使用此函数作为损失函数只需在编译模型时指定损失函数为focal loss
model.compile(loss[binary_focal_loss(alpha.25, gamma2)], metrics[accuracy], optimizeroptimizer) 参考
深度学习推荐系统作业deepFM介绍和代码实现_deepfm代码_J_co的博客-CSDN博客CTR算法总结_Roger-Liu的博客-CSDN博客DeepCTR模型优缺点对比_deepctr训练加速_ChristineC_的博客-CSDN博客deepFm的keras实现_deepfm keras_xxaxtt的博客-CSDN博客如何用keras实现deepFM_DemonHunter211的博客-CSDN博客最通俗的deepFM理解及keras实现_keras实现deepfm_Mr_不想起床的博客-CSDN博客用Keras实现一个DeepFM_deepfm 基于libsvm实现_蕉叉熵的博客-CSDN博客DeepCTR易用可扩展的深度学习点击率预测算法包 - 知乎python:sklearn标签编码(LabelEncoder)_sklearn labelencoder_jenny_paofu的博客-CSDN博客【推荐系统】DeepFM模型_deepfm多分类求auc_—Xi—的博客-CSDN博客GitHub - shenweichen/DeepCTR: Easy-to-use,Modular and Extendible package of deep-learning based CTR models .理解推荐算法之DeepFM_Aliert的博客-CSDN博客
